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ABSTRACT 

Designed to provide pre- and inservice vocational 
education administrators with background information and hands-on 
experiences for acquiring the skills necessary to direct a program 
evali^a^ion^ this competency- based learning module contains an 
introduction and four sequential learnina experiences. Each learning 
experience consists of an overview, reqii** ^d and optional learning 
activities, a self-check section, and a scriries of model answers for 
use with the self-check section. Topics covered ia the first learning 
experience are the purpose of program evaluation, evaluation methods, 
and cost ef f ectivess/benef it analysis of postsecondary vocational 
programs. Developing evaluation plans, preparing to impleaant an 
evaluation ?lan, and program evaluation skills for administrators are 
among the areas examined in the second learning experience. Discussed 
next are using evaluation data, presenting evaluation reports, and 
developing and evaluating conclusions and recommendations. The final 
learning experience involves actually directing a program evaluation. 
(Related competency-based vocational education administrator modules 
covering curriculum development, improving instruction, program 
promotion, staff development, and student recruitment and admissions 
are available separately through ERIC--see note.l (MN) 



Leadership and Training Series No. 58t3-10 



OO 
CO 

O 

a 

LU 

DIRECT PROGRAM EVALUATION 



COMPETENCY-BASED VOCATIONAL EDUCATION ADMINISTRATOR MODULE SERIES 



Consortium for the Development of Professional 
Materials for Vocational Education 

Robert E. Norton, Consortium Program Director 
James B. Hamilton, Consortium Associate Program Director 
Lois G. Harrington, Program Associate 
David R. Greer, Graduate Research Associate 
Karen M. Quinn, Program Associate 



The National Center for Research in Vocational Education 

The Ohio State University 
I960 Kenny Road 
Columbus, Ohio 43210 



ERIC 



1981 



U S DEPARTMENTOF HEALTH, 
EDUCATION & WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS BEEN REPRO- 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORIGIN- 
ATING IT POINTS OF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRE" 
SENT OfTFICiAL NATIONAL INSTITUTE OF 
EDUCATION POSITION CR POLICY 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



The work presented herein was performed by the National Center 
for Research in Vocational Education on behalf of the Consor- 
tium for the Development of Professional Materials for Voca- 
tional Education. Sponsors and members of the Consortium for 
1978--1979 included the following states and/or cooperating 
agencies: the Florida Department of Education, Division of 
Vocational Education, and Florida International University, 
Division of Vocational Education; the Illinois State Board of 
Education, Department of Adult, Vocational, and Technical Edu- 
cation, and Southern Illinois University at Carbondale; the 
New York State Education Department, Office of Occupational and 
Continuing Education; the North Carolina Department of Public 
Instruction, Division of Vocational Education; the Ohio State 
Department of Education, Division of Vocational Education; the 
Pennsylvania Department of Education, Bureau of Vocational Edu- 
cation, and Temple University, Department of Vocational Educa- 
tion; and the Texas Education Agency, Division of Occupational 
Education. The opinions expressed herein do not , however , 
necessarily reflect the position or policy of any of the spon- 
sors, and no official endorsement by them should be inferred. 



These materials may not be reproduced, except by members of the 
Consortium, without written permission from The National Center 
for Research in Vocational Education. 



FOREWORD 



The need for competent administrators of vocational edu- 
cation has long been recognized. The rapid expansion of voca- 
tional education programs and increased student enrollments have 
resulted in a need for increasing numbers of vocational adminis- 
trators at both .the secondary and postsecondary levels. Preser- 
vice and inservice administrators need to be well prepared for 
the complex and unique skills required to successfully direct 
vocational programs. 

The effective training of local administrators has been 
hampered by the limited knowledge of the competencies needed 
by local administrators and by the limited availability of 
competency-based materials specifically designed for the prepara- 
tion of vocational administrators. In response to this pressing 
need, the Occupational and Adult Education Branch of the U.S. 
Office of Education, under provisions of part C — Research of the 
Vocational Education Amendments of 1968, funded the National 
Center for a scope of work entitled "Development of Competency- 
Based Instructional Materials for Local Administrators of Voca- 
tional Education" during the period 1975-77. That project had 
two major objectives: 

1. To conduct research to identify and nationally verify 
the competencies considered important to local admin- 
istrators of vocational education. 

2. To develop and field test a series of prototypic 
competency-based instructional packages and a user's 
guide. One hundred sixty-six (166) high priority 
competencies were identified and six prototypic modules 
and a user's guide were developed, field tested, and 
revised . 

While six modules had been developed, many more were needed 
to have competency-based materials that would address all the 
important competencies that had been identified and verified. 
In September 1978 several states joined with the National Center 
for Research in Vocational Education to form the Consortium for 
the Development of Professional Materials for Vocational Edu- 
cation. Those states were Illinois, Ohio, North Carolina, 
New York, and Pennsylvania. The first five states were joined 
by Florida and Texas later in the first year. The first objec- 
tive of the Consortium was to develop and field test additional 
competency-based administrator modules of which this is one. 

Several persons contributed to the successful development 
and field testing of this module on directing curriculum devel- 
opment. Lois G. Harrington, Program Associate, assumed the major 



responsibility for reviewing the literature and for preparing 
the actual manuscript. Special recognition also goes to 
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INTRODUCTION 



Occupational technology is in a state of constant change. 
Rising costs have accentuated citizens' concern with how their 
tax dollars are being spent. State and federal agencies want to 
be sure that special populations are receiving a quality educa- 
tion in the least restrictive environment. Vocational educators 
want to know if their graduates are being placed and are succeed- 
ing in their chosen occupations. In order to meet all these con- 
cerns, program evaluation is essential. 

Program evaluation, as shown in the figure that follows, is 
a continual process. One can evaluate a single occupational spe- 
ciality (e.g., welding), a vocational service area (e.g., trade 
and industrial education), the total vocational program within 
the educational institution, the district vocational program, or 
the district educational program as a whole. Whatever the scope 
of the program to be evaluated, the evaluation process does not 
start with data collection or end with the publication and dis- 
semination of an evaluation report of findings. Evaluation 
starts with a need to answer some questions or make some deci- 
sions concerning a program. Data collection is designed to pro- 
vide those answers or that data for decision making. Thus, an 
evaluation includes using that data to make decisions — to make 
changes. This, in turn, requir<^:s the completion of additional 
evaluation efforts in order to determine the effectiveness of the 
decisions and changes made — and so the process continues its 
unending cycle. 

In today's world of educational specialization, it is not 
uncommon for the responsibility for the various components of the 
evaluation process to be divided. For example, an c dministrator 
may define the scope and purpose of the evaluation. An evalu- 
ation specialist may plan the methodology, select or develop the 
necessary instruments, and conduct the actual evaluation effort. 
A group of selected staff may use the results to brainstorm for 
interpretations, conclusions, and recommendations. Final evalu- 
ation reports for various audiences may be prepared by public 
relations staff. And finally, the process flows back to the 
administrator, who needs to make decisions or changes based on 
the evaluation findings and recommendations. The strength of 
this shared effort is founded on the fact that it should yield 
more objective results, since the evaluators are specialists with 
no vested interest in the program(s) being evaluated, with no 
need to provide justification for program effectiveness. 

On the other hand, however, it is very easy for a shared 
effort to deteriorate into a fragmented effort, with evaluation 
methodology and data unrelated to the key questions raised 
initially, or with evaluation results treated as an end in 
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themselves and never being translated into action, change, or 
decision making. This highlights the absolute necessity for 
active involvement of the- administrator in the total process. 
If you as an administrator understand the whole process and 
regard your responsibility for its total management as important, 
then you will "keep a hand in" throughout-~-leading , encouraging, 
directing, monitoring, suggesting, and doing your share. Only 
then will evaluation be a unified effort with usable results. 
Only then will evaluation result in action. 

This module is designed to pro'^ide you with background 
information on the evaluation process and hands-on experiences 
that will help you acquire the skills you need to define your own 
administrative role in evaluatic"* and effectively direct a voca- 
tional program evaluation. 




Module Structure and Use 



This module contains an introduction and four sequential learning 
experiences. Overviews, which precede each learning experience, 
contain the objectives for each experience and a brief descrip- 
tion of what the learning experience involves. 



Objectives [. Terminal Objective ; While working in an actual 
{•administrative situation, direct program evalu- 
tation. Your performance will be assessed by your 
I resource person us ing the "Administrator Perfor- 
^mance Assessment Form," pp, 75-77, (Learning 
I Experience IV) ^ 

fcr . . ^ • ........ 

Enabling Objectives ; 

1. After completing the required reading, demon- 
strate knowledge of the need for, and impor- 
tant considerations involved in, directing 
program evaluation. (Learning Experience I) 

2. After completing the required reading, develop 
two evaluation plans based on the information 
provided in given case situations. (Learning 
Experience II ) 

3. After completing the required reading, plan 
how to use the evaluation data provided in a 
given case study. (Learning Experience III) 



Resources a list of the outside resources that supplement 

those contained within the module follows. Check 
with your resource person (1) to determine the 
availability and the location of these resources, 
(2) to locate additional references specific to 
your situation, and (3) to get assistance in 
setting up activities with peers or observations 
of skilled administrators. 

Learning Experience I 

Optional 

• REFERENCE; Kim, Jin Eun. Cost Effective - 
ness/Benefit Analysis of Postsecondary Voca - 
tional Programs . Indianapolis , IN; Indiana 
State Board of Vocational and Technical Edu- 
cation , 1977 . 



AN ADMINISTRATOR experienced in the direc- 
tion of program evaluation whom you can 
interview. 

REFERENCE: Alkin, Marvin C. et al. 
Evaluation and Decision Making; The Title 
VII Experience , Los Angeles, CA: Univer- 
sity of California, Center for the Study of 
Evaluation, 1574. 

REFERENCE: Combs, Arthur W. "Educational 
Accountability from a Humanistic Perspec- 
tive." Educational Researcher . 2 (Septem- 
ber 1973): 19-21* 

REFERENCE: Macdonald, James B. "An 
Evaluation of Evaluation." The Urban 
Review. 7 (January 1974): 3-14. 

REFERENCE: Provus, Malcolm. Discrepancy 
Evaluation: For Educational Program 
Improvement and Assessment . Berkeley, CA: 
McCutchan Publishing Corporation, 1971. 

REFERENCE: Scriven, Michael. "The Meth- 
odology of Evaluation." In Perspectives of 
Curriculum Evaluation . Chicago, IL: Rand 
McNally, 1967. 

REFERENCE: Stake, Robert. "The Counte- 
nance of Educational Evaluation." Teachers 
College Record . 67 (April 1967): 523-540. 

REFERENCE: Phi Delta Kappa National 
Study Committee on Evaluation (Daniel L. 
Stuff lebeam et al . ) . Educational Evalu - 
ation and Decision Making . Itasca, IL: 
F. E. Peacock Publishers, 1971. 

REFERENCE: Tyler, Ralph. Basic Princi - 
ples of Curriculum and Instruction . 
Chicago, IL: University of Chicago Press, 
1949. 

REFERENCE: Worthen, Blaine R. , and 
Sanders ^ James R. Educational Evaluatio n : 
Theory and Practice" Worthing ton, OH: 
Charles A. Jones Publishing Co., 1973. 

A GROUP OF PEERS with whom you could com- 
plete the optional discussion question. 
(The activity can also be completed on an 
individual basis. ) 
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Learning Experience II 



Optional 

• REFERENCE: Program Evaluation Skills for 
Busy Administrators , Portland , OR: North- 
west Reg lonal Educational Laboratory, 1977 . 

• REFERENCE: Morris, Lynn Lyons. Program 
Evaluation Kit. Beverly Hills, CA: Sage 
Publications, 1978. 

• REFERENCE: Cook, Desmond L. Program 
Evaluation and Review Technique: Applica - 
tions in Education . Cooperative Research 
Monograph No. 17. Washington, DC: U.S. 
Department of Health, Education, and Wel- 
fare, 1966. 

Learning Experience III 

Optional 

• REFERENCE: Morris , Lynn Lyons , and 
Fitz-Gibbon, Carol Taylor. How to Present 
an Evaluation Report . Beverly Hills, CA: 
Sage Publications , 1978 . 

• REFERENCE: U.S. Department of Health, 
Education, and Welfare; Office of Education. 
Preparing Evaluation Reports: A Guide for 
Authors. Washington, DC: U.S. Government 
Printing Office, 1970. 

Learning Experience IV 

Required 

• AN ACTUAL ADMINISTRATIVE SITUATION in 
which, as part of your duties, you can 
direct program evaluation. 

• A RESOURCE PERSON to assess your compe- 
tency in directing program evaluation. 



Selected Terms Administrator — refers to a member of the second- 
ary or postsecondary administrative team. This 
generic term, except where otherwise specified, 
refers to the community college president, vice- 
president, dean, or director; or to the secondary 
school principal, director, or superintendent. 

Board — refers to the secondary or postsecondary 
educational governing body. Except where other- 
wise specified, the term "board" is used to refer 
to a board of education and/or a board of 
trustees. 

Institution — refers to a secondary or postsec- 
ondary educational agency. Except where other- 
wise specified, this generic term is used to refer 
synonymously to secondary schools, secondary voca- 
tional schools, area vocational schools, community 
colleges, postsecondary vocational and technical 
schools, and trade schools. 

Resource Person — refers to the professional educa- 
tor who is directly responsible for guiding and 
helping you plan and carry out your professional 
development program. 

Teacher/Instructor — these terms are used inter- 
changeably to refer to the person who is teaching 
or instructing students in a secondary or post- 
secondary educational institution. 

User's Guide For information that is common to all modules, 

such as procedures for module use, organization of 
modules, and definitions of terms, you should 
refer to the following supporting document: 

Guide to Using Competency-Based Vocational 
Education Administrator Materials ^ Columbus , 
OhI The Center for Vocational Education , The 
Ohio State University, 1977. 



This module addresses task statement numbers 19-22 and 25 from 
Robert E. Norton et al.. The Identification and National Verif i- 
ca tion of Competencies Important to Secondary and Post-Secondary 
Admini strators of vocational Education (Columbus, OH; The Center 
for Vocational Education, The Ohio State University, 1977). The 
166 task statements in this document, which were verified as 
important, form the research base for the National Center's 
competer>cy-based administrator module development. 
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Learning Experience I 



OVERVIEW 




Activity 



Optional 
Activity 



^ Optional 
^Activity B 




After completing the required reading, demonstrate 
knowledge of the need for, and important consider- 
ations involved in , d ir ecting program evaluation • 



You will be reading the information sheet, "Pro- 
gram Evaluation: What Is It? Why Do It? How Is 
It Done?" pp. 11-22. 



You may wish to read the supplementary reference, 
Kim, Cost Effectiven ess/Benefit Analys is of Post - 
secondary Vocational Programs . 



If at all possible, arrange through your resource 
person to meet with an experienced administrator 
and interview this person concerning his/her 
experiences in directing program evaluation. 



You will be demonstrating knowledge of the need 

for, and important considerations involved in, 

directing program evaluation by completing the 
"Self-Check," pp. 23-24. 



continued 
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OVERVIEW_con^inu£d . 



YOU will be evaluating your competency by compar- 
ing your completed "Self-Check" with the "Model 
Answers," pp. 25-26. 




Optional 
Activity 



you may wish to review selected readings describ 
ing existing evaluation models and to respond to 
related discussion question on a small-group or 
individual basis. 
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For information defining what program evaluation is^ 
why it Is important, and, in general, how it is done, 
read the following information sheet. 



PROGRAM EVALUATION: WHAT IS IT? WHY DO IT? HOW IS IT DONE? 



Evaluation Defined 

The evaluation of vocational education programs has been 
important since the beginning of vocational education. However, 
its importance has been amplified with changing financial and 
economic conditions and with more prescriptive legislation than 
ever before experienced • With these changes in the import of 
evaluation have also come changing meanings and connotations for 
the term. Evaluation to some has been thought of as student 
assessment, and to others, faculty assessment. To still others, 
evaluation has brought forth the thought of accreditation through 
regulatory visitations to schools and programs. A broad and 
workable definition of evaluation is as follows: 

Evaluation is the collection of information and judgments 
from a wide variety of sources to facilitate planning, to 
aid in the improvement of programs, and to meet account- 
ability demands. 

This definition provides an umbrella under which many data- or 
information-gathering activities can be placed, and emphasizes 
evaluation as a positive, constructive activity. 



Purposes of Evaluation 

There are many reasons for evaluating vocational programs. 
These vary from meeting external demands to providing information 
useful to the continuation , enhancement , and improvement of the 
offerings to students. The following discussion elaborates some 
of the purposes of vocational program evaluation and answers the 
question, "Why evaluate vocational programs?" 

To improve programs . Of key importance in the definition 
previously presented is the improvement of programs. This is 
probably the most important purpose that evaluation can fulfill. 
The collection of evaluative information and its use in making 
programs better are much like the quality control and quality 
assurance functions in business and industry. Evaluation can 
help to ensure that instruction is relevant and current, and that 
planning decisions about program improvement are based upon the 
best available facts and figures. 
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TO assist in ma king and justifying decisions. decisions 
that are based upon intuit ion alone often lead to undesirable 
outcomes, either'^through actions taken or P^l^ ^^^^ ^^"ff^I 
Rational decisions, baled upon evaluative information, are defen 
sibie aid justiciable to pr^ram staff, institutional administra- 
tors and supporters of the Vocational program. Evaluation can 
^r^vide information that assists in making decisions about such 
matters as assignment of personnel, selection of students, pro 
gram changes, budget allocations, and others. 

TO meet a ccountability demands . Evaluation can provide 
evide nce t^ indicate whethL the outcomes °f ^ "°rr 
the investment. This requires more than a financial summary or 
audit. It necessitates the presentation °f f "^^^^ ^ost 

?e a Placement results, achievement scores) in relation to cose 
nJo;i;a?ion ?he audience for accountability reports "^^^J^J 
UD of the supporters or ov-rseers of the program. It might 
Tnflude Administrators, boards of control, advisory committees, 
and other community leaders. 

TO promote or p ublicize vc <:ational programs. Another pur- 
pose 7§F ^eva?uating vocationa -Tp^ograms relates to the need to 

the communicltion of balanced information can provide a valuable 
JehicJe for gaining and maintaining support for a program as well 

for recruiting 'students. By making P-ple aware of the suc- 
cesses achieved by vocational programs, as well as the snort 
cominqs thatthe institution is working to overcome, reports of 
eSaiCItions can help maintain the credibility and advance the 
image of vocational education. 

TO meet State and federal man dates. ^^"^^^^^^^^^i^^i^ 
requirements stemming from education ^^^^^^^f^ion federal and 
stlte regulations, and government agency i^"^^ * "f^i^'IJ^ 

state and federal mandates is often perceived to be of JJ^^o^ 
f^,?,ortance or utility to local school personnel; in fact, man- 

ifvel^^ThrSducation -end.ents of 1976 man ate t^^^^^ 

fvfrf flve^yeTr^'^-^strterhLf pa^ed this rLpLsibility on 

:r!^cfredLation a.e-ies (LEAs) The P^^^^ 

^h^ndlcapp^ed^^a^ TncrlLe^ the °n f --^^afa ^'^"^^^^ " 

document their efforts in these areas with aard data. 
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Shortcomings of Traditional Evaluation Attempts 



There are many reasons why evaluation has not been used to 
its fullest potential in meeting one or more of the purposes 
elaborated in the previous section. It is important to present 
some of these reasons so that current-day evaluators can design 
evaluation activities and systems that avoid or overcome these 
pitfalls or undesirable effects. 

Evaluation has been misused . Evaluation results have sel- 
dom been used formatively — to make changes and improvements in 
programs. Often, evaluation has been used to show what or who 
was in error, without providing paths for future improvement. 
Follow-up reports, reports of student achievement, and faculty 
evaluation results are not always used for developing plans for 
improvement or change. Additionally, accreditation and state- 
directed evaluation reports are, at times, viewed as the end of 
an effort rather than the beginning of improvement planning. It 
must also be admitted that sometimes evaluation has set out to 
prove that a project or program was successful — whatever the 
data. 

Evaluation has been mandate-oriented . Mandates for evalu- 
ating vocational education programs, as mentioned in the previous 
section, exist at numerous levels. When evaluation is done sim- 
ply to meet mandates, it is sometimes done in the easiest and 
most expeditious way, with little concern for its use in meeting 
other purposes. The student follow-up reporting demands placed 
upon states have sometimes been met without concern for using the 
information internally, or for collecting other information at 
the same time that might be useful to states and local education 
agencies. 



Evaluation has been threatening . Much evaluation has 
focused on finding the skeleton in the closet or identifying the 
persons who aren't pulling their share of the load. This puni- 
tive focus has caused people to become fearful and anxious about 
any form of evaluation. Even when evaluation has a positive 
improvement purpose, this fear can still present an obstacle to 
success unless L,iven proper attention. 



Evaluation ha^; lacked commitment . Evaluation in vocational 
education has not always received the amount of financial and 
personnel resources commitment needed to optimize its impact. 
When compared to the investments made in quality control within 
manufacturing and service industries, educational institutions 
have only begun to make adequate investments in the evaluation of 
vocational progress — too often it has had only token support. 

Evaluation com p etency has been missing . One obvious reason 
that evaluation has not been used to its fullest potential is 
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that administrators so.eti.es lac. -npetency^in ^l^^^, 

rmirneedert^'desig'n ^valuaUon aSti'vities properly, nor have 
tSey been able to implement effectively those ^?;;%P^°^i^f 
others! This module has been developed primarily to help you 
develop these important competencies. 

Involvement of Significant Groups 

The evaluation of vocational programs, to be successful and 
efficIentrLst involve -P-^^.^^^.i^f .Jr^vriuluon ff^o^rhas 
^^:^ibiu?rforv'ar!iSs c'on^erned audiencL!^ that important 
expert se is included, and that the evaluation results are used 
expertibt i. and their representatives might play dif 

purposes outlined previously. 

instr uctors ^achers . Instructors should be involved in 

real change. 

Adv isory committee/council . Advisory ^°'";;i"^^/^°""^^^3e 
members have made a commi Uue.i L "to vocational education. These 

demand studies. 

A^^.;r,-!c:»-rai-ors YOU as an administrator, as well as admin- 
• . ?^^r^t all SI her levels, must be involved in evaluation 
istrators at all other leve, initiating evalu- 

efforts. Even If y°^/"^°^t :nd sanction is usually important 
ation activities, your support a effort. In addition, you 

to the implementation of the -al^^^.^^ ^^^^^^^ ^ liaison with 
can act as an a dvisor to T.n« monitor of progress:!^ AdHTnistra- 
rrra^a^?^ev%lrwi^rplt?'a^f^^ in 'sd, th. evaluation 
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results; therefore, it is important that they be committed and 
involved. 



Support personnel . Other personnel such as counselors and 
placement personnel often have potential for contributing to an 
evaluation. In addition to having evaluative opinions, these 
individuals will many times have expertise and information that 
is necessary to the process of evaluation. For example, coun- 
selors or placement personnel are usually in an excellent posi- 
tion to assist in or direct a student follow-up survey. 

Students . For many evaluation activities, students are the 
most knowledgeable concerning actual course and program opera- 
tions. Students have more contact time with instructors and some 
support personnel than any other group mentioned. Therefore, 
their input is critical to an evaluation. Also, students can be 
involved in the preparation or revision of evaluation instruments 
such as follow-up questionnaires. For example, having students 
react to a follow-up questionnaire during its development may 
alleviate language and interpretation problems that future 
respondents might have with the questionnaire. 

Community organizations . Some organizations you may wish to 
involve in the evaluation are labor unions, industry councils, 
professional associations, parent groups, and chambers of com- 
merce. Also, public agencies such as CETA, employment service/ 
security commissions, rehabilitation services, and the Veterans 
Administration can be useful in providing information from their 
perspective about crucial parts of the environment of vocational 
education. That is, they represent sources of support and are 
"users" of the "products" of the institution. 

External e valuators . In addition to representatives from 
the previously mentioned groups, it is often beneficial, although 
costly, to involve evaluation experts from private consulting 
firms, universities, or other agencies. External evaluation per- 
sonnel can play several roles. First, they can be used to assist 
in the design and implementation of an internal evaluation sys- 
tem, beginning with the identification of evaluation needs on 
through the use of evaluation results. A second role involves 
using an external person as a director of an external evaluation. 
For example, many special and innovative activities funded by 
state and federal agencies require that a neutral third-party 
evaluator be contracted. A third-party evaluator is one who is 
external to the the funding agency and the institution (first and 
second parties). The external evaluator, in this role, conducts 
the evaluation and reports to both parties. (See sample 1 for 
specific suggestions on how to secure an outside evaluator.) 

External evaluators who may assist you in facility evalu- 
ation at no cost include persons from the Environmental 
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SAMPLE 1 



''^.^rPQ^IONS FOR SECURING A THIRD-PARTY 
SUGGESTIO^NS FOR CONSULTANT 



specify evaluation-purpose and scope. 

available for the evaluation. 



Determine 



financial resources a^ 



Prepare a summary 
Proposal IRFP] ) • 



of needed assistance (may 



be a Request for 



Xdentify potential evaluators or evaluation consultants. 



Distribute work statement or summary 
potential evaluators. 



ry of needed assistance to 



Review prospecti or proposa 
select the two or three best proposals. 



is submitted by evaluators. 



iscuss their offers and 



Meet with the top two °^ ^^5'^;^^^^°ous work, 
to review examples of their prev 

^h.f have employed the evaluators. 
contact former agencies that have emp y 

Select the best one. 

^^t-vaf-t- to finalize details. 
. prepare an agreement or contract to _ , 



Protection Agency (EPA) or the Occupational Safety and Health 
Administration (OSHA) , fire marshals, boiler inspectors, and 
others concerned with safety. 



Focus of Evaluation 

The previous sections of this information sheet have men- 
tioned a number of focuses for evaluation such as faculty per- 
formance, student performance, and others. This section will 
specify additional focal points that may be appropriate in the 
evaluation of vocational programs. 

The Education Amendments of 1976 (Title II) highlight four 
areas of evaluation that must be addressed by state boards of 
education in their evaluation of vocational programs. These 
include (1) planning and operational processes involving facili- 
ties, equipment, services, and curriculum, (2) results of stu- 
dent achievement as measured by competency tests, (3) results of 
student employment success (placement and wages), and (4) other 
results as measured by services to special populations. 

Additional focal points to consider in evaluating vocational 
programs can be identified through the review of standards or 
criteria used by accreditation agencies and state education agen- 
cies. For example, one state has conducted extensive research to 
identify the important components of a successful total program 
of vocational education. These components are (1) program man- 
agement, (2) planning and evaluation, (3) community resources, 
(4) program content, (5) personnel, and (6) student services. 
The evaluation of a total vocational program might focus on all 
of these components. 

If an evaluation is to be more specific, focusing on a 
single course or program, it might consider the following points: 
(1) students served, (2) goals and objectives, (3) organization, 
(4) personnel, (5) content, (6) teaching methods, (7) learning 
achievement, and (8) supplies, facilities, and equipment. Obvi- 
ously, the focus of the evaluation will depend upon the purpose, 
scope, and term of program. Equally important, the evaluation 
purpose and scope will dictate the program elements that are to 
be analyzed or evaluated. 



Evaluation Techniques or Processes 

There are many techniques or activities that can be used 
obtain evaluative information, formulate judgments, and assist 
improving programs. A detailed description of each potential 
technique is beyond the scope of this module. However, a brie 



description is provided, and you are referred to the list of 
references in sample 2 for additional information. 



Student follow-up survey . This activity is designed to help 
staff gather data concerning the instructional programs and ser- 
vices from former students. Additionally, the student follow-up 
can help to determine the effectiveness of programs in terms of 
job placement — a product evaluation measure. A mailed question- 
naire, telephone interview, or personal interview can be used to 
obtain opinions, ratings, and suggestions from former students. 
The method should be chosen based on the number of students to be 
surveyed, their geographical dispersion, and the funds and number 
of personnel available. Be particularly wary of the notoriously 
low rate of response often resulting from nail-back question- 
naires. The survey should be based upon the need for specific 
program improvement information that will aid in answering broad 
key questions formulated by staff. It should be planned and 
coordinated by a team of individuals who represent various groups 
on the staff. This activity can focus on the total institution, 
its vocational program, or individual programs or courses. 

Employer follow-up survey . This activity is designed to aid 
in assessing the on-the-job performance of former students. It 
involves contacting employers to obtain their ratings and com- 
ments on the performance of former students. The employer survey 
can be conducted using a mailed instrument, a telephone inter- 
view, or personal interview. Regardless of the method chosen, 
the information gathered should answer prespecified key questions 
that are formulated by staff. A local leader and a team of staff 
and advisory committee members should plan and coordinate the 
survey. This activity can focus on the total institution, a 
vocational service area, or individual programs or courses. 

Student interest survey . This activity is designed to 
facilitate the collection and use of student career interest 
information. This may involve the use of standardized interest 
inventories or locally developed instruments. Interest informa- 
tion is used in two ways: (1) to aid in student career planning, 
and (2) to aid in curriculum and program planning. Teachers, 
counselors, curriculum specialists, and students should be 
involved in this activity. The activity may be conducted for 
the total student body or certain instructional levels, or indi- 
vidual instructors may choose to use it independently within 
their classes • 



1. For additional information on follow-up surveys, you may wish 
to refer to Establish a Student Placement Service and Coordinate 
Follow- Up Studies, a Competency-Based Vocational Education Admin- 
istrator Module (Columbus, OH: The Center for Vocational Educa- 
tion, The Ohio State University, 1977). 
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SAMPLE 2 



RECOMMENDED SOURCES FOR EVALUATION INFORMATION, 
TECHNIQUES, AND INSTRUMENTS 



Albright, L. ; Evans, JU ; and Fabac, J, An Identification and Assessment Syatem for Special Needs Learners . Culde 
No. 8: "Student Follow-up-" Guide No. 9; "Employer Survey." Urbana, IL: University of Illinois, 1978. 

Afih, L. C. Instrupents and Procedures for the Evaluation of Vocational/Technical Education . Waahlngton, DC; American 
Vocational Association, 1972. 

Babble, Earl R. Survey Research Methods . Belmont, CA: Wadsworth Publishing Company, Inc., 1973. 

Backstros, Charles H. , and Hursh, C, D. Survey Research . Evanston, IL; Northwestern University Press, 1963. 

Bloom, B. S. ; Hastings, J. T. ; and Madaus, G. F. Handbook on Formative and Sunaatlve Evaluation of Student Learning . 
New York, NY: McGraw-Hill Book Company, 1971. 

Borgen, J., and Davis, D. Planning, Implementing, and Evaluating Career Preparation Programs . Bloomlngton, IL: 
McKnlght Publishing Company, 1974. 

^ Boyd, J. L., Jr., and Shlaberg, B. Handbook of Performance Testing . Princeton, HJ; Educational Testing Service, 1971. 

Byram, Harold M. , and Robertson, Maivin, cocnps. and eds. Locally Directed Evaluation of Local Vocational Education 
ProRratrs: A Manual for Administrators^ Teachers, and Citizens . Third Edition. Danville, IL: The Interstate 
Printers & Publishers^ Inc., 1971. . , 

Darcy, Robert L. Vocational Education Outcomes: Perspective for Evaluation . R&D Series No. 163. Columbus, OH: The 
National Center for Vocational Education, The Ohio State University, 1979. ' • , 

Edwards, A. L. Techniques of Attitude Scale Construction . New York, NY: Appleton-Century-Crof ts , 1957. 

Erlckson, R. C. , and Wentllng, T. L. Measuring Student Growth: Technlquea and Procedures for Occupational Education . 
Boston, MA: Allyn & Bacon, Inc., 1976. 

Franchak, Stephen J., and Splrer, Janet E. Guidelines and Practices for Follow-Dp Studies of Former Vocational Educa- 
tion Students, Vol. 1 .' R 4 D Series No. 171. Columbus, OH: The National Center for Research In Vocational 
Education, The Ohio State University, 1979. • . . 

Franchak, Stephen J., and Splrer, Janet E. Guidelines and Prac t ices for Follow-Up Studies of Special Populations, 

Vol. 2. R&D Series No. 172. Columbus, OH: The National Center for Research tn Vocational Education, The Ohio 
State University, 1979. •; : .. 

Glrod, G. R. Writing and Asseaalng Attltudlnwl Objectives . Columbus, OU: Charles E. Merrill Publishing Co., 1973. 

Gronlund, N. E. Constructing Achievement Tests . Englewood Cliffs, NJ: Prentice-Hall, Inc., 1968, 

Headrlck, M. L. Making Effective Uae of Follcv^Up Information . Pensacola, FL: The University of West Florida, 
Department of Technical and Vocational S;udlet;, 1977. " .,. 

Illinois Office of Education. Three Phase System for Statewide Evaluation of Occupational Programs; Team Leader 
Handbook . Springfield, IL: State Board of Education, n.d. 

Irvln, D. E. , and Russo, R. Procedures Manual for the Minnesota Secondary Follow-up System . Minneapolis, MN: Univer- 
sity of Minnesota, 1978. 

Kester, Ralph. Using Systematic Observation Techniques in Evaluating Career Education . R&D Series No. 169. 
Columbus, OH: The Center for Vocational Education, The Ohio State University, 1977. - 

McCaslln, N. L. ; GroBS» Charles; and Walker, J. P. Career Education Measures; A Compendium of Evaluation Instruments . 
R &*D Series No. 166. Columbus, OH: The National Center for Research in Vocational Education, The Ohio State 
University, 1979. ^ V .- • 

McKlnney, F, L. , and Oglesby, C. Developing and Conducting Follow-up Studies of Former Studf^nts . Lexington, KY: 
Kentucky Research Coordinating Unit, 1971. . - * ' 

The National Center for Research In Vocational Education. Career Education Measurenent Series; Assessing Experiential 
Learning In Career Education ; Career Education Measures; A Compendium of Evaluation Instrumenta ; A Guide for 
Improving Locally Developed Career Education Measures ; Improving the Accountability of Career Educat ion Programs; 
Evaluation Guidelines and Checklists ; Using Systematic Observation Techniques In Evaluat ing Career Education. 
Colurebus, OH: The National Center for Research in Vocational Education, The Ohio State Unlveralty, 1979. 

Ohlo^ Department of Education. PRIDE: Ohio Program Review for Improvement, Development, and Expansion In Vocational 
Education and Guidance— Procedure Guide for the VEPD Coordinator . Columbus, OH; Department of Education, 197b. 

Stufflebem, Daniel L. , and Webster, William J. "An Analysis of Alternative Approaches to Evaluation." Educational 
Evaluation and Policy Analysis . 2 (May-June 1980): 5-20. ^ * ■ ' ' -* •. 
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, Evaluation of instructional materials . This activity, 
focusing on tne evaluation ot the LEA's collection of instruc- 
tional materials, may include three phases that are desiqned to 
improve the collection and use of materials. These Phasis fre 
(1) the inventory of all instructional materials thai are owned 
by the institution, (2) an assessment of the adequacy of the 
and tin ^n ^"'^ utilization of the results of^ases one 

and two to increase the use of materials and also to aid in the 
fni materials. A combination of methods— includ- 

ing the survey, staff rating of materials, and student rating 
of materials--is possible. A local leader (possibly a media 
director or librarian) should be appointed to direct a team of 
to^fo- •"'^ support personnel in this effort. The activity 

IS designed primarily for assessing the institution's entire Col- 
lection of instructional materials, although it can be adapted to 
focus on certain segments of the collection, such as the audio- 
visual media available. 

Team evaluation. This activity involves the use of a team 
of educators; business, industrial, and labor representatives- 
and former students to analyze the total vocational program o^ 
its various components. This review process involves two teams 

^nH ^h^'' J\ ^°.?^^" activity and prepare materiJTs, 

and the second to actually conduct the review. The review team 
observes interviews, and studies available information !n an 
attempt to formulate conclusions about the program and to suggest 
ways of improving identified deficiencies. The purpose of the 
review team is to provide consultant assistance that includes 
suggestions for improvement. This activity can be applied to the 
total institution, a vocational service area, or individual pro- 
grams and courses. ^ 

Student testing. This activity, in which all instructors 
are already involved, stresses the use of student test data in a 
program evaluation. The activity involves the development of' 
instruments and measurement techniques to assess the achievement 
of various forms of student performance objectives within the 
cognitive, affective, and psychomotor domains. It also involves 
the preparation of individual and group profile forms that can be 
helpful in using the results. Instructors should assume the pri- 
mary responsibility for completing this activity, and cooperation 
among instructors who teach the same courses or subjects should 
be encouraged. The activity can be done by all instructors° or 
individual instructors can use it independently. ^^""^s, or 

Evaluation of community resources . This activity is 
designed to aid in the evaluation of the availability and effec- 
tiveness of community resources such as prospective advisorv 
committee members, guest speakers, and field trips. The acLvitv 
has two parts. Part one focuses on the identification of aJaU-^ 
able resources through the use of a mailed questionnaire or a 
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telephone survey. The result of part one is the establishment 
of a resource file or catalog. Part two focuses on the rating 
of the effectiveness of these resources by students and staff 
through the use of a printed questionnaire or rating scale. The 
results of part two can be incorporated into the resource file 
and used to improve the selection of outstanding resources. A 
team of individuals — consisting of administrative staff, instruc- 
tors r and advisory committee members — should coordinate the 
activity. If possible, this activity should be applied to the 
total LEA; however, it can be applied to any instructional com- 
ponent of the institution. 

Student evaluation of instruction . This activity is 
designed to aid staff in obtaining information from current stu- 
dents to facilitate the improvement of classroom instruction. A 
team of staff generally develops a questionnaire or rating form 
to be administered to currently enrolled students to determine 
instructional effectiveness. The questionnaire results are then 
summarized, and each instructor (with support from others) pre- 
pares a report that includes suggestions for improving the 
instructional process. This activity can be conducted for the 
total institution, vocational program, individual programs, or 
individual courses. Emphasis is placed on the use of results by 
individual instructors. 

Evaluation of facilities and equipment . This activity is 
designed to help in analyzing the adequacy of existing facili- 
ties/equipment for the purpose of determining future needs for 
expansion, renovation, or abandonment. Additionally, this 
activity can help in increasing the use and improving the safety 
of facilities. This activity involves the inventory of facili- 
ties and the observation and rating of facility characteristics 
by both advisory committee members and staff members. A local 
leader and a team of internal staff should plan and coordinate 
the evaluation and should prepare a report with recommendations . 
This activity can be applied to the entire institution, the 
vocational program, or individual programs and courses. 

Cost/Outcome analysis . This activity is designed to aid 
personnel in collecting information regarding costs and relating 
it to information about outcomes. The activity can focus on the 
cost/outcome relationship of (1) a program, (2) a course, (3) a 
unit of instruction, and (4) a type of media. The results of 
the activity can assist in making decisions about instructional 
alternatives and can help increase the efficiency of instruction. 
One part of the activity involves the collection of cost infor- 
mation from instructional staff members and existing records. 
Outcome information is gained from existing measures, e.g., test 
scores, follow-up results, other ongoing evaluation activities, 
and special data-collection procedures. This activity can be 
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applied to the total institution, the vocational program, occu- 
pational specialties, individual courses, or more specific com- 
ponents. 



For more information on the cost-effectiveness/ 
Optional benefit analysis model, you may wish to read Kim, 
i Activity M Cost Effectiveness/Benefit Analysis of Postsecondary 
\ ^ Vocational Programs , This document provides an 

overview of the model and a. manual of the specific 
activities needed to conduct an evaluation in this 



area. 



It is strongly recommended that you arrange through 
your resource person to meet with and interview an 
Optional administrator with successful experience in directing 
Activity ■ program evaluation. Before the interview takes 

§^ place, you should prepare a list of questions you 
wish to have answered, e.g. : 

• How directly is he/she involved in each step 
of the evaluation process? 

• What problems has he/she encountered and how 
can they be avoided or solved? 

• What techniques does he/she find to be most 
effective? 



EKLC 
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Activity 



The following items check your comprehension of the 
material in Ihe information sheet, "Program Evalu- 
ation: What IS It? Why DO It? HOW Is It Done? 
pp. 11-22. Each of the items requires a short 
essay-type response. Please respond fully, but 
briefly, to each item. 



SELF-CHECK 



this essential? 



0 If an administrator directs an exemplary Program evaluation 
effo^tfresulting in a sound evaluation report, is his/her 
duty fulfilled? Why or why not? 



3. HOW involved should administrators be in the total evaluation 
process? 



23 



How important is the involvement of others (instructors/ 
teachers, advisory committee/council members, students, other 
administrators, support personnel, etc.) to the success of 
the evaluation effort? 



Considering all of the evaluation techniques discussed in the 
information sheet, which technique is the best? 
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Compare your written responses on the "Self-Check" 
with the "Model Answers" given below* Your answers 
need not exactly duplicate the model responses; how- 
ever, you should have covered the same major points. 



MODEL ANSWERS 



It is crucial that data be gathered from a wide variety of 
sources in order to ensure valid, usable results. For exam- 
ple, assume you surveyed only former secondary students, 
and your results showed that (1) few students were employed 
in related fields, and (2) most felt they had been poorly 
prepared. With only this data from this one source, you 
cannot be 100 percent sure that this is the case. Community 
survey data or follow-up data from employers may indicate 
that poor preparation per se is not the problem. Rather, it 
may be that there have been few job openings in the area (or 
even cutbacks), or employers may have been hiring postsec- 
ondary graduates because of their more advanced training. 
Collecting information and data from a variety of sources 
allows you to compare responses and be more secure in your 
final interpretations . 

The preparation of the final evaluation report is not the 
end; it is the beginning of a new cycle in the evaluation 
process. Although the report does serve the purpose of docu- 
menting program accountability, if that is all it does, then 
the effort was largely wasted. For evaluation to be worth 
the time and resources involved, the results need to be used 
to improve programs and to assist in making and justifying 
program decisions . 

The extent to which administrators are involved in the total 
evaluation process will vary depending on their exact admin- 
istrative role and the size of the total administrative 
staff. A principal in a small, rural school may have total 
responsibility for performing each task in the process. A 
vocational director in a large, suburban district may dele- 
gate much of the responsibility to evaluation specialists on 
the staff. The point is that program evaluation is a key to 
program success; therefore, whether you develop your own 
evaluation plan or assign the task to others, you, as an 
administrator at whatever level, have to take an active 
interest in the process. You must ensure that you are kept 
abreast of evaluation activities on a continual basis. More 
over, delegation does not mean the end of responsibility. As 
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a decii3ion maker, you must monitor the process and make your- 
self available for assisting those to whom you have assigned 
direct responsibility. 

4. Involving others in the evaluation process is essential to 
the success of the evaluation efforts. No matter how excel- 
lent the administrative staff is, its view is somewhat 
limited. To design appropriate questionnaires for use with 
employers, input from advisory committee members and instruc- 
tors is invaluable. To identify potential problem areas that 
should be focused on in the evaluation, instructors and stu- 
dents must be involved. Involvement of these various groups 
can improve the planning efforts and usability of results 
greatly. More pragmatically, these persons are a potential 
source of needed workers during data-collection activities. 
Further , involving interested , concerned parties throughout 
the evaluation process improves the chances that recommenda- 
tions resulting from the evaluation will be accepted and 
implemented. Finally, those involved in evaluation often 
become more personally interested in the institution or pro- 
gram — "buying into it," so to speak. 

5. There is no one best evaluation technique. In undertaking an 
evaluation effort, the "best" techniques are the ones that 
best match such factors as your evaluation needs, the extent 
of your evaluation effort, the size of your sample, the 
resources you have available, and the questions you wish to 
have answered. First, one makes decisions about all these 
elements; then, one selects the most appropriate technique (s) 
accordingly. 



Level of Performance ; Your completed "Self-Check" should have 
covered the same major points as the model responses. If you 
missed some points or have questions about any other points you 
made, review the material in the information sheet, "Program 
Evaluation; What Is It? Why Do It? How Is It Done?" pp. 11-22, 
or check with your resource person if necessary. 
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Optional J 
Activity ^ 



If you wish to gain a greater understanding of the 
theoretical constructs underlying educational evalu- 
Optional \ ation, it is strongly recommended that you complete 
Activity ^ the following activity. The activity requires that 
some basic reading covering differing theories and 
points of view be done, and that this reading be used 
to address a discussion question that is provided 
concerning evaluation. The purpose of this activity 
is not only to familiarize you with the existing 
evaluation models, but also to help you to develop 
your own rational stance toward evaluation — one that 
incorporates the best from all that exists and 
resolves conf 1 ic ting views . 

The activity can be structured in a variety of ways. 
If you are completing this module at the same time 
as peers (e.g., as part of a university course), you 
could complete this activity as a group. For exam- 
ple, you could each study one model or point of view 
and then address the discussion question through 
debate. Or, you could each complete all the readings 
and address the discussion question in a seminar ses- 
sion. If peers are not available to you, you could 
complete the reading and respond to the discussion 
question in writing. Your resource person could be 
asked to attend any debates or seminars, or to review 
and discuss with you the adequacy of your written 
response . 



Reading ; 

Using the references listed below (and cited on p. 6), 
others listed on the inside back cover, and others you can 
locate on your own, become familiar with at least four evalu- 
ation models:^ 

1 . Goal-Attainment Models 

Tyler, Metfessel, Michael, and Hammond 
( see Tyler and Pro v us references ) 

2. Judgmental Models Emphasizing Extrinsic Criteria 
Scr iven' s format ive/summative evaluation 
Stake's Countenance Model 

(see Scriven and Stake references) 



2. Delineation of four models and related references were drawn 
from Basic Concepts in Vocational Education , Module 13 in the 
Vocational Education Curriculum Specialist (VECS) Series (Palo 
AltOr OA: American Institutes for Research, 1976), pp. 23-28. 
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3. Judgmental Models Emphasizing Intrinsic Criter ia 
Accreditation Model 

(see Worthen and Sanders reference) 

4. Decision-Facilitation Models 
Alkin's CSE Model 

Provus' Discrepancy Model 
Stufflebeam's CIPP Model 

(see Alkin et al., Provus, and Stuff lebeam et al. 
references) 

In addition, become familiar with the points of view offered 
by humanists James Macdonald and Arthur Combs, critics of the 
recent emphasis on educational evaluation (see Macdonald, Combs, 
and Stufflebeam references). 



Discussion Question ; 

In order for evaluation to be truly objective, valid, and 
reliable, it must be systematized and quantifiable. The evalu- 
ation models are an attempt to provide such systems. Yet, 
critics point out that they see flaws in each system. Measur- 
ing success by identifying goals and then assessing the degree 
to which those goals are attained makes sense only if the right 
goals have been identif ied--not an easy task. The Accreditation 
Model considers the adequacy of in-place resources (facilities, 
staff, instructional materials, and equipment), but largely 
ignores the worth of what is taught. Use of isolated evaluation 
specialists can separate the collection and presentation of data 
from the reality of educational goals and decision making, thus 
potentially limiting the usefulness of the data. Complicating 
this is the idea that when one evaluates the success of a newly 
installed innovation, one assesses it on its own terms (i.e.. Did 
it do what it was supposed to do? Is it still in place and oper- 
ating as planned?) Yet, this ignores the fact that adaptation is 
often a part of the assimilation process and that — although the 
innovation may seem to disappear after two years — a teacher or 
program may operate quite differently as a result of exposure to 
that innovation. 

Given this framework, respond to the following five discus- 
sion questions: 

1. What, briefly, are the steps and premises defining each 
of the four models? 

2. What do you see as the strengths and weaknesses of each 
model? 

3. Are the criticisms offered by Macdonald and Combs valid? 
Why or why not? 
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What evaluation model (an existing one, an adaptation of 
one or more existing models, your own model) would you 
use to ensure that the right questions are asked con- 
cerning the right goals and that the data is both poten- 
tially useful and actually used? 

What do you perceive to be the role of vocational admin- 
istrators in the evaluation process? How extensively 
should they be involved? How can they ensure continu- 
ity? 
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Learning Experience II 
OVERVIEW 




After completing the required reading, develop two 
evaluation plans based on the information provided 
in given case situations. 



Activity 



( 

V 



Optional 
Activity 





You will be reading the information sheet, "Devel- 
oping and Preparing to Implement an Evaluation 
Plan," pp. 33-47. 



You may wish to read one or more of the supple- 
mentary references, Program Evaluation Skills for 
Busy Administrators ; Morris, Program Evaluation 
Kit; and/or Cooky Program Evaluation and Review 
Technique . 



You will be developing two evaluation plans based 
on the hypothetical situations described in the 
"Case Situations," pp. 49-50. 



You will be evaluating your competency in develop- 
ing evaluation plans by comparing your completed 
plans with the "Model Plans," pp. 51-53. 



EKLC 
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Activity 



For information describing the steps involved in 
developing an evaluation plan and preparing to imple 
ment that plan, read the following information sheet 



DEVELOPING AND PREPARING TO IMPLEMENT AN EVALUATION PLAN 



Establish a Team for Planning and 
Coordinating the Evaluation Effort 

The first task addresses one of the necessary features of 
effective evaluation systems — involvement , A team of individual 
should be selected to aid in the design, implementation, and use 
of the program evaluation system. if you as administrator are 
directing the total evaluation effort, the selection of such a 
team is your responsibility. The team will vary in both composi 
tion and size with the focus and scope of the evaluation. How- 
ever, it is generally advantageous to involve a cross section of 
individuals who will be affected by the evaluation, or contacted 
during the conducfof its activities. It is also good to involv 
individuals who will contribute willingly and who are highly 
respected by others. Involvement of the right people in the 
planning phase can enhance the design and acceptance of the 
evaluation ef f ott . 

In selecting a planning and coordinating team, a necessary 
first step is to determine the desired representation. The 
groups that might be included are administrative personnel, 
instructional personnel, learners, ancillary personnel, advisory 
committee members, and external experts. Common sense should be 
used in deciding on representation. Factors to be considered 
should include the focus of the evaluation, the scope of the 
evaluation, and the resources available to conduct the evalu- 
ation. Thus, a team with the task of designing an evaluation 
system for programs in a mul tiattendance secondary vocational 
center might have the following composition: 

1 Vocational dean or director 

2 Principals (one from each of two feeder schools) 
2 Department chairpersons 

4 Teachers (3 vocational, 1 general) 

1 Director of institutional research 

1 Counselor or placement director 

1 Chairperson of the vocational advisory council 
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1 Employer of program graduates 

2 Students (1 current, 1 former) 



A smaller institution might not have such a large team. Instead, 
a team might include the following: 

1 Vocational director or principal 
3 Teachers (2 vocational, 1 general) 
1 Counselor or placement director 
1 Advisory committee member 

1 Employer of program graduates 

2 Students (1 current, 1 former) 

If an individual department or program is the focal point of 
the proposed evaluation, of course, the team will be much more 
limited. 

A second step in establishing a team to plan and coordinate 
the evaluation is to select and invite individuals to comprise 
the team according to the composition that has been determined. 
In this step, you may wish to use your own knowledge of people, 
ask other administrators to suggest individuals, or solicit volun- 
teers. The local situation will dictate your method of selec- 
tion. Once they accept your invitation to serve, members should 
be officially appointed by letter, by board resolution, or both. 



Schedule and Hold a Meeting of the Planning Team 

Following the establishment of the planning and coordinating 
team, it is important that you promptly schedule a meeting for 
the purpose of initiating work on the actual design of the evalu- 
ation system. One of the first tasks is to identify the optimum 
time for such a meeting to ensure good attendance. 

Once the team is assembled, considerable effort should be 
made to orient this group to their task, as well as to explain 
why they were selected and your expectations for them in their 
role as planning and coordinating team members. Many times in an 
orientation meeting of this type, it is advantageous to have a 
high-ranking official or administrator deliver a statement to the 
team to set the pace. This can help all involved to develop a 
commitment to the evaluation. 

The planning and coordinating team should be informed that 
their overall responsibility is (1) to develop or adapt a system 
for evaluation that is designed to meet the needs of the particu- 
lar program or institution, and then (2) to prepare a plan of 
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action for implementing the proposed evaluation system. It 
should be stressed that this general responsibility will be 
shared by all and that, where possible, staff support and tech- 
nical assistance will be provided. 

It may also be necessary during this initial meeting to dis- 
cuss some of the evaluation practices that have previously been 
used by the institution. This may aid the group in identifying 
some of the strengths or weaknesses that past evaluations had 
and can set the stage for how future evaluation activities might 
evolve. In addition, it is important that early in the planning 
process the team consider statewide program standards and evalu- 
ation procedures that might apply. Information about standards 
and procedures is available from the state division or department 
of vocational education. You need to be sure that you collect 
the data that are required and report them in a form usable by 
the state department for its own information-gathering purposes 
and for comparing local data with that obtained from other 
institutions . 

The last major task of the initiation meeting is to outline 
the specific activities or tasks in which the planning and coor- 
dinating team will be involved. The remaining tasks in planning 
an evaluation system include the following: 

• Developing and formalizing a purpose and scope statement 
for the evaluation system 

• Developing key evaluation questions 

• Selecting appropriate evaluation activities 

« Identifying leaders for each activity (frequently, you 
as administrator may serve as evaluation leader) 

• Sequencing evaluation activities 

• Presenting the evaluation components in the form of a 
written evaluation plan 

Each of these tasks can be accomplished through meetings, discus- 
sions, and independent work. Your involvement in the completion 
of these tasks may vary from direct involvement to delegation of 
the responsibility to others on your administrative or teaching 
staff. Regard?^ess, since the entire evaluation effort hinges on 
decisions and plans made at this point, you cannot afford to 
become too removed from the planning process. You will probably 
want to make substantial input into the development of the pur- 
pose and scope statement, the choice of key questions, and the 
selection of leaders. But even if instrument selection and the 
writing of the evaluation plan are assigned to others, you must 
conscientiously monitor their progress and carefully review their 
results so a solid plan is developed. 
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Develop and Formalize a Purpose and Scope 
Statement for the Evaluation System 

The first substantive activity in the design of an evalu- 
ation system involves determining and stating the purpose and 
scope of the planned evaluation system. The purpose for program 
evaluation may already have been determined by internal staff or 
by some administrative agency, or it may grow out of a need to 
make certain planning or management decisions. However, there is 
a need to involve the planning team in formalizing the purpose 
and stating it in writing. The purpose should succinctly state 
why program evaluation is being undertaken. This kind of state- 
ment, of course, can help in communicating the purpose to others 
and can aid in alleviating any anxiety or threat that might be 
posed by the word evaluation . 

Simultaneous or integral to the development of a purpose 
statement is the determination of scope or breadth for the evalu- 
ation effort. The scope statement, in essence, states what will 
be included in the evaluation. You may wish to have separate 
purpose and scope statements, or it is possible— of ten advanta- 
geous — to include both in one statement. 

Experience has shown that the evaluation leader (who may be 
you or someone you have designated) should first draft one or 
more statements of purpose and then request that team members 
react zo and revise this^raft. Using this approach can save 
time in accomplishing this task. 

The purpose statement in itself is very important to the 
total direction of the evaluation effort. Therefore, it is 
necessary to give considerable thought and attention to its 
preparation. Sample 3 presents several examples of purpose 
statements for program evaluation systems. It should be noted 
however, that these are just examples; the purpose statements 
for all evaluation systems will vary from program to program or 
institution to institution. 

The scope statement for the evaluation system is easier to 
develop than the purpose statement. The scope statement merely 
indicates which p .rt of the educational or training agency will 
be involved in the evaluation, e.g., total institution, voca- 
tional program, a single occupational specialty, or an individ- 
ual course. The following is an example of a scope statement: 

The evaluation will focus on the Associate of Arts program 
in dental hygiene. 

By combining and reviewing the purpose and scope statements, the 
general thrust for the evaluation effort can be determined. 
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SAMPLE 3 

PROGRAM EVALUATION SYSTEM STATEMENTS OF PURPOSE 



^ . To determine whether stated goals or objectives have been 

^ met. . ' 

(. • • • . • 

F ■• * ; ■ • ■ 

To determine what aspects of our total occupational program 
need improvement. 

To determine needed curriculum content revisions. 

To determine how well orientation courses prepare students 
\ for training-level courses. 

(• . 

i 

To learn what impressions occupational program graduates have 
j concerning the program that they completed, 

t ■ ■ • ■ • ■ 

i .*•■''■' 

1 • To determine staff inservice needed for program improvement, 

I . . ■ ' ■ ' * 

To improve the decision-making process used in planning occu- 
pational programs. 



To assess occupational supply-and-demand information in light 
of needed program additions and deletions. 

To provide sufficient data to aid the instructional and 
administrative staff, as well as the board, in their decision- 
making functions. 

To gather sufficient data to inform the public of outcomes 
resulting from educational expenditures. 

To aid in identifying areas that need expansion or removal 
within our occupational offerings. 

To determine what aspects of our program need the greatest 
allocation of resources for improvement. 

To broaden occupational offerings according to student needs. 
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Develop Key Evaluation Questions for the Syst em 
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Evaluation questions are broad questions that narrow the 
focus of the evaluation system from the purpose and scope state- 
ments down to the individual parts or segments of the component 
being evaluated. Evaluation questions take the purpose state- 
ment and make it specific. Evaluation questions reflect v;hat is 
important to know as a result of an evaluation, i.e., the ques- 
tions you expect to have answered by the evaluation. 

Activities, information sources, and instruments should be 
selected only if they help answer the evaluation questions. This 
places great emphasis on the need for care and attention to the 
development of evaluation questions. 

Program improvement needs may be the most valuable source 
for identifying key questions. General concerns or feelings of 
the staff may reveal some of these, but other sources should also 
be considered. State or administrative evaluation reports and 
regional accreditation findings can provide indications of pro- 
gram components that may need further analysis or evaluation. 

The following is an outline for you or another designated 
evaluation leader to follow in helping the planning and coordi- 
nating team formulate evaluation questions. These steps may be 
altered to fit the needs of individual agencies, 

1. Build and present a rationale for having key questions, 

a. To focus the evaluation 

b. To aid in selecting evaluation activities 

c. To aid in developing instruments 

d. To aid in using evaluation results 

2. Present possible sources of key questions. 

a. Previous evaluation reports ( accreditation » state 
visits , third-party) 

b. Concerns of staff 

c. Planning decisions and needs 

3. Secure sample lists of key questions (from this module 
or from other institutions) and distribute to the team, 
(See sample 4 for some examples.) 

4. Duplicate or obtain multiple copies of previous on-site 
evaluation reports (accreditation, state, etc.) or other 
informational document? that might help in identifying 
key questions . 

5. It may be advantageous if you, the evaluation leader, 
highlight some of your priority concerns from any of the 
three points listed under item 2. This may give the 
planning team a starting point. 
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SAMPLE 4 
SOME KEY EVALUATION QUESTIONS 



• Do students enrolled in our vocational programs learn the 
basic cognitive skills of speaking, reading, writing, and 
numerical calculation as effectively as comparable students in 
academic/general curricula? 

• Do our vocational education programs provide graduates with a 
general set of skills and basic work disciplines that signifi- 
cantly increase their chances for successful employment? 

• How do the annual earnings of our vocational graduates com- 
pare with those of general/academic graduates, both in the 
short term and in the long run? 

• Do our vocational programs significantly strengthen the human 
resource base of the local and state economy? 

• Does favorable cominunity reaction to ongoing vocational pro- 
grams result in general expressions of satisfaction and finan- 
cial support? 

• Are the vocational students developing leadership ability 
through their participation in occupat ionally related youth 
programs such as DECA, FHA, FFA, and VICA? 

• Do our vocational programs increase occupational awareness , 
training opportunities, and job access for men and women in 
both traditional and nontraditional areas? 

e Do our vocational education program admissions and student 
assignment practices tend to reduce or to reinforce occupa- 
tional discrimination on the basis of race , class , or sex? 



SOURCE: Adapted from Robert L. Darcy, Vocational Education Out - 
comes; Perspective for Evaluation (Columbus , OH: The National 
Center for Research in Vocational Education, The Ohio State Uni- 
versity, 1979), pp^ 15-19. 
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6. Depending on the size of the planning team, as a group 
or as divided subgroups, write some potential key ques- 
tions (keeping in mind the overall purpose and scope 
statement). You may wish to explain this task at one 
meeting and then ask the team members to bring some 
questions to the next meeting. 

7. Later in the meeting or at the next meeting of the team, 
review several questions and begin discussing each one. 

8. You will probably have more questions than you can 
feasibly answer within the resource limits for evalu- 
ation. Initially, it is better to have a long list of 
questions^ some of which you can give priority to later. 
Don't be afraid to adapt questions from the examples 
provided, but don't rely on them entirely. Remember, 
the evaluation system that you are designing should meet 
your specific needs. 

9. Duplicate the list of key questions developed by the 
team. If the list is long, obtain team reaction about 
priority questions. 

10. Prepare final list of key evaluation questions. 



Select Appropriate Evaluation Activities 

Based upon the selected evaluation questions, it is neces- 
sary to determine (1) what data already exist that can be tapped, 
and (2) what additional evaluation techniques or activities can 
assist in providing answers. As shown in sample 5, these include 
interviews, questionnaires, document review, testing, and obser- 
vation. (Information on more specific techniques is included in 
Learning Experience I, pp. 17-22.) It is helpful to use the 
staff's past experience with certain evaluation activities as a 
source of input in selecting activities. Another alternative is 
to use the services of an evaluation expert or consultant to 
assist in the selection of evaluation activities. 

In selecting activities (e.g., employer survey), it is 
essential that those in charge of this Process (you^^other staff , 
or 
eaci 

tils of each*activity, the focus" for each evaluation activity 
should be redefined. Familiarity with each activity may leaa to 
either a broadening of the focus of the evaluation questions or 





lead to a more efficient plan 



SAMPLE 5 
DATA-COLLECTION TECHNIQUES 



Interviews can be structured with a form or 
interview schedule that specifies the questions 
to be asked of each respondent. Interviews can 
also be open-ended, with the use of a topical 
outline that guides the interviewer in knowing 
what areas should be probed. The interview can 
be conducted face to face or by telephone. 

Questionnaires are printed forms designed to 
collect information and judgments from respon- 
dents. They can include many different item 
types, including checklists, graphic rating 
scales, multiple-choice items, numerical rat ing 
scales, matching items, etc. Questionnaires 
can be administered by mail or distributed per- 
sonally. In either case, a strong follow-up 
effort is essential in order to secure an ade- 
quate , representative response . 

Document review involves the analysis of 
already existing information that may be of 
secondary use. It can be done informally to 
estimate general feelings or in a very formal 
way using checklists or other structured 
instruments . 

The administration of student tests can provide 
valuable information. Testing can focus on 
cognitive, affective, and/or psychomotor behav- 
iors. Instruments used in testing can include 
paper-and-pencil tests, inventories, simulation 
tests, performance tests, projective tests, and 
many other devices. 

Observation can provide descriptive information 
regarding the way something is constructed or 
behaves. Observation can be unstructured, or 
it can be structured through a special record- 
ing form. Observation can be either open, 
secret, or visually recorded. 
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Identify Leaders for Each Activity 



The members of the planning team or you, as an administra- 
tor, usually will not be able to work directly with all aspects 
of each evaluation activity. Therefore, it is important for you 
to appoint a person (or several persons) to serve as leaders of 
each important part of the evaluation process. Some of the same 
principles that applied to the selection of planning team members 
will also guide the appointment of these leaders. That is, they 
should be committed to the goals of evaluation, be familiar with 
the subject of the evaluation, be able to locate information and 
resources needed to fulfill their duties, be well versed in 
evaluation principles, and be respected by others. 

Furthermore, these leaders must possess many of the usual 
administrative skills, such as the ability to organize and com- 
plete assignments, work effectively with colleagues, manage time 
and resources, and express themselves well orally and in writing. 
In most cases, you will expect the leaders to determine their own 
methods for accomplishing the activities assigned them by the 
evaluation planning team; considerable liberty can be allowed 
them with respect to procedures, but the time sequence and dead- 
lines should be adhered to closely. 



Sequence Evaluation Activities 

Once evaluation activities have been selected, it is impor- 
tant to develop a calendar or sequence chart for the evaluation 
system. This calendar should include an indication of the plan- 
ned initiation and completion dates for each of the selected 
evaluation activities. In scheduling, it is often valuable to 
carefully review resources that provide detailed descriptions of 
how to conduct each of the selected activities. Through review, 
the major tasks and subtasks of each activity become clear, and 
time estimates can be made. A review may also make evident which 
activities might precede others. For example, by conducting a 
student follow-up study, the names of employers may be solicited 
from former students. Therefore, the employer survey might most 
efficiently follow the former student follow-up survey. 



P resent the Evaluation System in the 
Form of an Evaluation Plan 

After the purpose and scope statement, and evaluation ques- 
tions are determined f and evaluation activities are selected and 
scheduled, all of this information should be transferred to a 
brief evaluation plan form. Sample 6 is an example of an evalu- 
ation plan form that includes categories covering the desired 
information. Once completed, such a plan can be useful in 
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SAMPLE 6 

r EVALUATION PLAN FORM 



• Purpose : 



I Scope 



* Key Evaluation Questions: 



: Information Sources: 



Data-Collection Techniques: 



■ Schedule of Events 



Activities 



Leader(s) 



Target Dates 
BegTn End 
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communicating the outline or structure of the evaluation system 
to everyone. It also provides a general monitoring schedule for 
you or another evaluation leader. 

You will need to identify those persons or groups (e.g., 
board, higher-level administrators) whose sanction and approval 
of the plan are required. Copies of the plan should then be 
submitted to these persons for approval. The approved evaluation 
plan should be distributed to all members of the planning and 
coordinating team, as well as to key individuals in the education 
or training program under consideration. 

Publicity is a factor here, too. It is helpful to arrange 
for the evaluation plans — once approved — to be announced using 
local media (newspaper, radio, TV). Community awareness of the 
need for and purpose oiE the evaluation effort can help you both 
when you attempt to secure their cooperation during data collec- 
tion, and later, when you attempt to make decisions and changes 
based on the evaluation results. 



Pr epare to Implement Evaluation Plan 

Each activity listed on the final plan can be further broken 
down into subtasks. For example, if your plan includes conduct- 
ing a student follow-up survey, the subtasks would probably 
include the following: 

1. Selecting appropriate persons or groups for involvement 
and assistance (e.g., advisory council/committee, stu- 
dents, teachers, administrators, counselors, board 
members ) 

2. Determining the best follow-up procedure to use 

3. Stating the objectives of the follow-up study 

4. Identifying the former students to be studied 

5. Designing cover letters, questionnaires, follow-up 
letters 

6. Designing interview guidelines 

7. Developing a schedule of activities with time speci- 
fications and ass ignments 

8. Conducting the survey 

9. Keeping records of survey progress 

10. Summarizing, interpreting, and reporting the findings 

11. Implementing approved recommendations 
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By listing each subtask and then describing it in detail, you can 
project the necessary resources and determine who should be 
responsible for or involved in each subtask. For example, as 
administrator, you might wish to take responsibility yourself 
for chairing the group development of an evaluation plan. How- 
ever, you may wish to delegate data collection to evaluation 
specialists. 

Staff assignments should, of course, be made for each of the 
subtasks. The scope and extensiveness of the evaluation activity 
will dictate the number of individuals needed. The director of 
the evaluation should probably assume overall leadership of all 
subtasks, though with extensive activities it may be advanta- 
geous to also identify individual leaders for each of the sub- 
tasks (see p. 42). The director can ask for volunteers in 
situations where no special knowledge is needed or can select 
individuals when special expertise is required to complete the 
job. 

The next step involves the sequencing of individual subtasks 
in a configuration that is most efficient in terms of resource 
and time utilization? i.e., developing an evaluation schedule. 
Many activities can be conducted simultaneously, but others will 
have to be conducted in a definite sequence. To initiate this 
sequencing step, all of the subtasks should be listed, and those 
that are dependent on preceding tasks should be identified. 
Then, the activities can be placed into a pictorial schedule or 
PERT-type chart such as the one in sample 7 in which subtask 1, 
2, and 3 must be completed before subtask 4 is begun, and work on 
subtask 2 can be started concurrently with work on subtask 1. 

Once activities have been scheduled and sequenced, esti- 
mates of time should be attached to each subtask. The example 
of sequence scheduling presented in sample 7 shows personnel 
assigned to each of the four activities and projected completion 
dates for each activity. It is often useful to include the names 
of those involved. Each of the subtasks may require more then 
one individual, and the name of at least the leader of the par- 
ticular activity, or the person responsible for its compleixcnr 
should be attached to the schedule. 

You will need to ensure that program personnel and all 
others who will be involved are (1) oriented to the purposes and 
the procedures of the evaluation activity, and (2) monitored 
throughout the data-collection and summarization processes. It 
is not enough to merely assign personnel to a given task. These 
people should clearly understand the thrust of the evaluation so 
they can make good decisions as the need arises, adequately 
explain evaluation goals to others, and comprehend the diffi- 
culties that might be met by other personnel. If all members 
are properly informed, they will be in a position to help one 
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SAMPLE 7 

GRAPHIC PORTRAYAL OF EVALUATION ACTIVITY SEQUENCING 



October 15 



December 1 



START 




Subtesk 1 




(Jonet) 






February 1 



END 
$ubta$k»2.3_ 

" "star" ~ 

Subtask 4 
I Fritz) 



J 



June 1 
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another. This is essential to the success of the evaluation 
itself. 

Finally, once the complete evaluation plan has been devel- 
oped, you will be ready to implement the evaluation program 
itself. It is not necessary for this module to cover this opera- 
tion in much detail— you have already come up with a comprehen- 
sive plan, so all you need to do is to follow it. You should 
pay particular attention to the schedule of activities you drew 
up and the relationships of crucial activities such as those 
depicted in the form shown in sample 7. By being continually 
aware of progress in terms of these deadlines, you can detect 
when additional resources (time, money, staff) might be needed to 
ensure that key activities are completed as planned. 

The planning and coordination team should continue to meet 
periodically during the implementation phase, primarily to moni- 
tor progress and detect and resolve unforeseen problems. You 
should help team members recognize that the original plan devel- 
oped was probably not perfect in every respect, so that they can 
understand the need to modify it in light of changing circum- 
stances. Most of the adjustments are likely to be in the areas 
of activity schedules or data-collection techniques. In most 
cases, you will retain the original purposes and key evaluation 
questions that guided your planning. Assuming that your Plan- 
ning was comprehensive, needed adjustments were made in a timely 
fashion, and your follow-through was thorough, then the data- 
gathering phase will be successful. 
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The Northwest Regional Educational Laboratory has 
available a training system entitled Program Evalu - 
ation Skills for Busy Administrators ^ This basic 
manual contains s ix two-hour learning modules that 
cover (1) defining the purposes of evaluation, 

(2) describing a program or activity to be evaluated, 

( 3 ) establishing evaluation guidelines , ( 4 ) preparing 
an evaluation plan, (5) implementing the evaluation, 
and ( 6 ) reporting evaluation findings . (Topics 5 
and 6 are covered in Learning Experience III of this 
module.) The manual may be purchased and used inde- 
pendently, or your resource person could obtain a 
Workshop Leader's Guide and offer the training in a 
group workshop format. 

The Program Evaluation Kit, a set of eight very 
practical guides written by and for evaluators, was 
developed at the Center for the Study of Evaluation 
at University of California at Los Angeles. The 
guides cover the following topics : ( 1 ) e valuator ' s 
handbook, including detailed, step-by-step direc- 
tions and practice activities for conducting a for- 
mative evaluation , a summative evaluation , or a 
small experiment; (2) how to deal with goals and 
objectives; (3) how to design a program evaluation; 

(4) how to measure program implementation; (5) how 

to measure attitudes; (6) how to measure achievement; 
(7) how to calculate statistics; and (8) how to pre- 
sent an evaluation report. (Topics 7 and 8 are 
covered in Learning Experience III of this module.) 
You may wish to review any or all of the eight guides 
in this kit for further, more detailed information on 
program evaluation. 

You can learn more about drawing PERT charts, esti- 
mating completion times, and sequencing activities by 
reading Cook, Program Evaluation and Review Tech - 
nique; Applications in Education . This book can 
help you to identify critical paths and events, cal- 
culate "slack time," and determine when schedules 
must be changed or resources reallocated to meet 
deadl ines . 
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All types of agencies can benefit from evaluation* 
The following "Case Situations" briefly describe two 
educational situations, each requiring the develop- 
ment and implementation of an evaluation plan*, Read 
the situations described, and develop an evaluation 
plan for each, designed to meet the evaluation 
requirements of the situation. Because you are 
dealing with sketchy hypothetical situations, your 
plans need not be very detailed; however you should 
cover each of the major sections to be included in 
such a plan. 



CASE SITUATIONS 



Case Situation 1 ; 

The occupational program at Sunbelt Community College is 
experiencing a decrease in enrollment, despite the fact that the 
surrounding area is generally prosperous and expanding economi- 
cally. Sunbelt was established 15 years ago and has a fairly 
comprehensive program in the areas of industrial education, 
business and office education, and health occupations. A recent 
program review was made, and it was determined that the programs 
offered do indeed meet the employment needs of the community. 
The various aspects or features associated with a community 
college include facilities, location, instructional personnel, 
administration, instructional program, public relations, and 
others. The month is now April. 



1 
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Case Situation 2 ; 

Metro Heights Vocational School serves several "bedroom 
community- su^Srbs of a large metropolitan ^-3;^^-%-^^°' 

the residents of the area ^^^J'J^^^^^^ "^J^'^here? Th^ of 
important part of the lives of ^he people there. in 

tmpllmented It the beginning o£ the next acaaemic year. 





Compare your written evaluation plans with the "Model 
Plans" given below. Your plans need not exactly 
duplicate the models; however ^ you should have 
covered the same major points. 



MODEL PLANS 



Case Situation 1 : 

Purpose: To stabilize or increase enrollment 

in the occupational program of a 
community college 

Scope: Public Relations Department 

Instructional Personnel 



Key Evaluation Questions: What public information services are 

currently being used? 

How do students learn about the 
Sunbelt program? 

Are students satisfied with the 
instruction they are getting? 

For what reasons do students drop 
out or fail to complete their pro- 
grams? 

How can information services be 
improved? 

Information Sources: Current Students 

Existing Records 
Former Students 

Data-Collection Techniques: Interviews with current students 

Student evaluation instrument 

Review of existing records 

Questionnaire survey of former 
students 



Schedule of Events: 



Activities 


Leader (s) 


Target 
Begin 


Dates 

End 


Interview current students 




May 1 


May 10 


Administer evaluation 
instrument to students 


*** 


May 15 


May 20 


Review records 




June 1 


June 10 


Survey former students 




July 7 


July 30 


Analyze and summarize 
findings 


*** 


Aug • 1 


Auu. 30 


Prepare report and dis- 
seminate findings 




Sept. 1 


Sept. 3 


Implement recommendations 




Oct. 1 


June 30 



Case Situation 2: 



Purpose: 



To revise curriculum, equipment/ and 
recruiting to better fit new tech- 
nology 



Scope : 

Key Evaluation Questions: 



Automobile mechanics program 

Is the field of automotive repair 
and service expand ing? 

What new skills and equipment must 
entry-level mechanics become famil- 
iar with? 

Are current graduates of this insti- 
tution adequately prepared to use 
these skills and equipment? 



Information Sources : 



Service managers of local automotive 
shops 

Instructors of auto mechanics 
Trade publications and associations 
Recent graduates 
Advisory committee members 



Data-Collection Techniques: Interviews with selected auto shop 

managers and mechanics 

Questionnaire survey of last four 
graduating classes 

Review of trade literature 



Schedule of Events: 

Activities Leader(s) Target Dates 







Beg 


in 


End 


Review trade literature 




Nov. 


1 


Nov. 


12 


Interview shop managers 


unit 


Nov 


15 


Dec. 


10 


Survey recent graduates 




Nov. 


15 


Dec . 


20 


Analyze and report findings 




Dec. 


20 


Jan. 


15 


Discuss results with 
faculty and advisory 
committee 




Jan. 


15 


Jan. 


30 


Implement approved 
recommendations 


nidi 


Feb. 


1 


Aug. 


30 



Level of Performance ; Your completed evaluation plans should 
have included each of the required components of such a plan 
(e.g., purpose, scope, etc.) and should have covered the same (or 
similar) major points as the models. If you missed some points 
or components or have questions about any additional points you 
made, review the material in the information sheet, '•Developing 
and Preparing to Implement an Evaluation Plan," pp. 33-47, revise 
your plans accordingly, or check with your resource person if 
necessary. 



53 

c>5 



L3arning Experience ill 



OVERVIEW 



After com-^leting the required readinc, plan how to 
use the evaluation data provided in a given case 
study. 



/ou will be reading che information sheet, "Using 
Evaluation Data," pp. 57-67. 



You may wish to read one or more of the supple- 
mentary references, Morris and Fitz-Gibbon, How to 
Present an Evaluation Report ; and/or U.S. Office 
of Education, Preparing Evaluation Reports; A 
Guide for Authors. 



You will be developing a list of conclusions and 
recommendations based on the partial program 
evaluation results described in the "Case Situ- 
ation," pp. 69-70. 



You will be evaluating your competency in develop- 
ing a list of conclusions and recommendations by 
comparing your completed list with the "Model Con- 
clusions/Recommendations," pp. 71-72. 



, Activity 



For information on reporting evaluation results and 
using those results in program planning and decision 
making, read the following information sheet. 



USING EVALUATION 



DATA 



The use of evaluation results is as key a step in the evalu- 
ation process as any other. Without proper use of results, the 
most sophisticated data collection and judgments will be wasted. 
Therefore, this step should receive primary attention. 

The process of using results has been traditionally over- 
looked, misunderstood, or underutilized. For example, reports of 
sc.Tie well-conceived and well-conducted follow-up studies have not 
even reached those persons who may have had potential uses for 
their contents. In other cases, evaluation results have been 
written into a report and distributed to key individuals; how- 
ever, their use did not extend beyond simple distribution and 
review. To attain the primary goal of evaluation — the improve- 
ment of offerings to students — evaluation results must be incor- 
porated into the decision-making and planning processes of 
educational administrators, faculty, and support personnel. 

The use of evaluation results can be divided into three 
interrelated steps: (1) interpreting, (2) reporting, and 
(3) decision making. The following sections provide descrip- 
tive information and suggestions concerning each of these steps. 
Again, your role in this process may vary, but since it is you 
who must make decisions based on the interpretations made, you 
should either be directly involved at this point or assign 
responsibility for interpretation and reporting to staff with 
the necessary expertise. 



Interpreting 

The interpretation of findings from an evaluation represents 
an important step in the evaluation process * Interpretation can 
occur at many levels, involving many different individuals and 
groups. For example, evaluation results might be displayed in a 
local newspaper. Consider the following presentation s 

A :ecent study conducted by Yankee Ridge Community College 
indicates that 90 percent of their vocational program 
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graduates obtained employment upon ^o^^P^^^^f" °^„^';e'^3 
instruction. By program, the placement results were as 

follows : 

Dental Hygiene 

Agricultural Mechanics 95% 

Auto Mechanics 

Nursing 

Day-Care Assistance 60% 

that data. 

interpretation of evaluative ^ata/f^^^^f'thf rurpoL'^Saf ?o 
to the purpose of the evaluation. . "^rprSted to facili- 

improve programs, then the data should be ^^^erpreted t^ 
tate improvement and not something else. J"^^;P;^. ^he pro- 

act of explaining or telling the structured 
cess, when associated with evaluation reports, can be str^ 
around developing an answer to the question What do tnese 
mean?" 

The results or answers to this question can ^^^^J^^^^^^^^^^ 
forms: (1) conclusions, (2) discussion, and (3) answ^^^^ to y^ 
questions. Although the process f^jJ^^^J^^P^^^.^jLIon format is 
^^^obfbry%STosi%rpulTr 

rf1Clg^fnrrhrt^^irb;en^Earef up-^^ h^^^^ 
evaluative information. Therefore, it will be used 



here. 



in addition to asKing^What f -ese^^data -n';;;^evaluators 
and decision makers also need inquire wn 

tions of these data ^nd conclusions? ^^^f^^^t °lso include 
happens next?" Thus, ^he evaluation effort must^^ ^^^^^^^ 
strong, clear recommendations that are lounucu 
the evaluation activities. 

1 i. • Kr.<-h f'nnrlusions and recommendations can 

1^ f„^trp7etruSirro„:r:"is ^.^,......0. 
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involving a ranye of individuals who may have a stake in a par- 
ticular program or evaluation focus. This concept of involving a 
broad cross section of personnel is just as important at this 
stage in the evaluation process as it was in the evaluation plan- 
ning stages. Therefore, a logical first task in interpreting and 
writing conclusions and recommendations is to identify and assem- 
ble a group of individuals who can facilitate interpretation. 
You can assume the role of group leader or assign this responsi- 
bility to another qualified individual. Then, the following 
steps (or an adaptation of them) should be followed. 

Step 1; General discussion . The staff conference should 
start with a presentation of a general overview of the task to be 
accomplished. Any questions regarding the procedures that were 
used in gathering the evaluation information should be raised and 
resolved. Then, each group member should be given an opportunity 
to offer comments or general information regarding the evaluation 
findings. 

Ste^ 2: State a conclusion . Each group member should be 
asked to draw one positive conclusion based upon the evaluation 
results. It may seem a waste of time to state positive conclu- 
sions regarding exemplary parts of the program when most evalu- 
ations seem to focus on deficiencies. However, there are sound 
practical and psychological reasons for focusing on the positive 
side. Psychologically, the consideration of positive program 
attributes will reassure program staff of the balanced nature of 
the evaluation. Practically, the inclusion of exemplary conclu- 
sions in the evaluation report can lead to using the strongest 
capacities and capitalizing on the program's strengths to over- 
come its recognized deficiencies. During the second "round," 
the group members might be allowed to present negative conclu- 
sions or identify areas of weakness that require improvement. 
As each person in the group presents a conclusion, that person 
or a stenographer should record the conclusion. 



Step 3; Review initial conclusions . The group leader 
should then review each of the conclusions that were developed 
in step 2 and give his/her reactions to the conclusion. This 
step will help ensure a consistent format and style for formu- 
lating subsequent 'conclusions. The following questions can aid 
in the analysis of initially developed conclusions: 

1. Is there sufficient evidence to support the conclusion? 
Does the conclusion reflect the consensus of the group? 

3. Is the conclusion compatible with the major goal and 
purpose of the evaluation? 

4. Is the statement precise and does it specifically note a 
characteristic of the program? 
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step 4; Team development of conclusions and associated 
recominendations . The size of the group will dictate the method 
used in formulating a series of conclusions and recommendations 
that relate to the purposes of the evaluation. If small-group 
divisions are made, with each small group considering only one 
segment of the evaluation results (such as those pertaining to a 
single evaluation objective), the entire group should, at some 
point, be afforded the opportunity to react to all conclusions. 
This can be accomplished by designating a leader for each small 
group who will be responsible for presenting the group's deci- 
sions orally. To encourage reaction and discussion by the group 
it is good to disseminate conclusions and recommendations in 
written form also or to present them using an overhead projector 
They should be revised or accepted intact, depending upon the 
group consensus. 

Step 5; Edit conclusions and recommendations . The group 
leader or evaluation director should collect all conclusions and 
recommendations from small-group leaders. The leader should 
review them and reword those that are in need of correction. 
All statements should be made parallel in style before they are 
included in the printed evaluation report. Furthermore, the con 
elusion statements should refer to the data or processes leading 
to each conclusion; in similar fashion, the recommendations 
should be tied in closely with related conclusions. 

Step 6; Staff review of written conclusions and recommenda 
tions . Once the leader has edited the conclusions and recommen- 
dations, these should be duplicated and given to staff for their 
consideration. Staff members should be asked to review the docu 
ments and return the copy to the leader, with comments, by an 
established date. 

These six steps can provide the basis for interpreting a 
set of evaluative data. A final step should involve inclusion 
of these interpretations in the evaluation report. 



Reporting 

The reporting of evaluative outcomes or results is some- 
times thought of as a nonessential activity, especially if the 
evaluator is an administrator or a key individual in the 
decision-making or planning process. However, unless evaluation 
procedures and findings are documented, the credibility of an 
evaluation may suffer. Additionally, an evaluation report pro- 
vides a vehicle through which a variety of individuals may be 
involved in improving programs and services. 



The preparation of an evaluation report should be a 
straightforward process with a given purpose and scope in mind. 
In other words, the report should reflect the original evaluation 
plan. The report should not be an esoteric research report; it 
should be a tool for communicating the results of evaluation to 
those individuals or groups who can use them. 

Since there may be different groups with different back- 
grounds and different potential uses of evaluative information, 
it may be necessary to prepare different versions of an evalu- 
ation report. Regardless of how many groups there may be, a 
first step in reporting is the identification of audiences for 
the report. The purpose of the evaluation system will usually 
provide, either directly or indirectly, an indication of who the 
potential audiences might be. For example, if an evaluation sys- 
tem purpose was "to provide accountability information to fund- 
ing agency personnel," then the obvious audience is the funding 
agency personnel. If the purpose was "to improve the curriculum 
materials used in the dental hygiene program," then the likely 
recipients of th^ evaluative information are program faculty and 
curriculum or media personnel. 

Audience identification is important in reporting because 
the material included, language used, background information 
required, and other features need to be specifically designed to 
meet the needs and characteristics of each audience. If the 
results are to be reported to the general public, then the style, 
format, and vocabulary level might approach that used in Time p 
Newsweek, or a newspaper. If, on the other hand, the audience is 
made up of researchers or curriculum development specialists, the 
report might take on a more technical flavor. In either case, 
clarity is more important than "making an impression" with your 
use of jargon. 

Generally, the evaluation report should contain five 
sections or components: (1) purpose and scope of the evalu- 
ation, (2) description of the program or unit being evaluated, 
(3) description of the evaluation methodology used, (4) evalu- 
ative findings, and (5) conclusions and recommendations. Much 
of the information for these five sections can be derived or 
transferred from the original evaluation plan. 

The first section presents the purpose and scope statement. 
This can be taken directly from the evaluation plan. In addi- 
tion, it may be useful to include a brief introduction that sets 
the tone for the remaining parts of the report. The key ques- 
tions may also be presented to provide the reader with a more 
specific idea of the evaluative focuso 

The second section — program description — should provide the 
reader with an overview of the program being evaluated. This 
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should help the reader to understand the context, constraints, 

^nd resources associated with the program. ^^^^^^ 

include program goals, student Performance objectives, descrip 

tion of students, description of program staff, and other program 

characteristics. 

The third section of the report should present a detailed 
descripti on of the eva luation methods and instruments used. This 
description may bS taken, in part, from the evaluation plan, with 
additional details provided regarding the . instruments or forms 
used For example, if one evaluation activity was a follow-up 
su^v^v tL description might indicate that a questionnaire was 
mauJd'to 300 former students on May 1, 1980 and that an 80 per- 
cent reJ^nse rating was attained. Additionally, an overview of 
?he questionnaire content might be provided, or a sample copy 
could even be included. 

The fourth section of the evaluation report should present 
the findi- ngs or outcome s of the evaluation activities, including 
^ H.^i?av of the information that was collected. The way evalu- 
^t?ie^iesu?ts are po??rayed is somewhat dependent upon who will 
be Ihe primary Audience or readers of the report, ^his portrayal 
cln considered in two ways. The first involves the way in 

5h?ch th^Lformation is grouped ^^.-'^-f ' ^i^^^l^^^d. 

deals with the form or format in which the data are displayed. 

One type of data grouping involves the presentation of 
information (1) according to how it was collected, or (2) by 
respondergriips. For 'example, if an valuation included a 
student foUow-uD study, an employer survey, and ^survey ot 
SurrSnt students; the data could be grouped according to these 
populations . 

A spcond type of data grouping involves breaking evaluation 

or ri^ta-co] lection techniques could be grouped into these two 
aref' Furthe^ grouping ?ould be accomplished by taking each 
key question and displaying all data associated "^^h it. This 
assists the audience of the report in comparing, contrasting, and 
aggregating data that pertains to a single point. 

This approach—relating data to key questions— is the 
preferred aoproach. It emphasizes the use of information in 
?omu.ating answers to questions and also facilitates the inter 
p?etation Ld ultimate use of evaluative results. 
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There are many different ways of presenting or displaying 
evaluative information within either of the aforementioned 
approaches. Again, the audience should be a consideration in 
choice of format. The underlying criterion in choosing a display 
format should be the question "What is the best technique for 
communicating the results to the audience?" The potential dis- 
play formats include, but are not limited to, bar graphs or 
tables of percentages, frequency counts, and averages. (See 
sample 8 for examples of these formats. )3 

The fifth (and probably most important) section of the 
report sho"uTd~contain the interpretations—conclusions and recom- 
mendations—of the information displayed. This secti.cn will help 
the reader translate the numerous data and should make it clear 
concerning what, if any, action is called for as a result of the 
entire evaluation process. 

The actual conclusion and recommendation statements should 
already have been edited in accordance with step 5 of the group 
interpretation process (see p. 60). When preparing the report, 
you should also pay particular attention to the order and manner 
in which these statements are presented. For example, you would 
want to list positive conclusions (strong areas) before citing 
negative conclusions (weaknesses). Also, each conclusion 
describing an area needing improvement should be followed imme- 
diately by recommendations concerning how this problem could be 
remedied. (Arrange recommendations in order of importance or 
acceptability. ) 

Be sure tha*- each recommendation states a required action 
clearly and affirmatively. Don't say "Alumni affairs need to be 
considered"; rather, suggest a specific step such as "Form an 
alumni association and provide the necessary office and staff 
assistance." If one recommendation might not be feasible because 
of limited resources, list it first anyway and then provide an 
alternative, e.g., "Obtain talking-book versions of key textbooks 
for visually-impaired students; if sufficient funds are not 
available, secure paid or volunteer readers to assist students 
with these special needs." 



3. To gain additional skill in presenting evaluative informa- 
tion, you may wish to refer to Module A-3, Report the Findings 
of a Communi ty Survey , part of the Professional Teacher Education 
" Module ser ies produced by the National Center fcr Research in 
vocational Education (Athens, GA: American Association for Voca- 
tional Instructional Materials, 1978). 



ERIC 



63 



SAMPLE 8 
DATA-DisPLAY FORMATS 



1. Ratings of students' initiative by employers: 



High 

Average 

Low 



Females 



Males 



Bar Graphs 




Female 


Male 


High 


10% 


0% 


Average 


40% 


30% 


Low 


40% 


70% 


Percentages 




Female 


Male 


High 


|>MUWi 


1 


Average 






Low 






Frequency Counts 


Female _ 


2.5 




Average 






(High = 3, 


Low - 1) 


Male 


1.9 




Average 




Averages 
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Decision Making 



Evaluation in general, and specifically in vocational edu- 
cation, can be focused on improved decision making. Decision 
making should, however, be considered from its broadest point of 
view. That is, planning, program improvement, and resource 
allocation are all forms of decision making. Therefore, they 
represent uses of evaluative information. 

Stufflebeam et al.^ have outlined four types of decisions 
in education: (1) planning, (2) programming, (3) implementing, 
and (4) recycling. 

Planning decisions are those that focus on needed improve- 
ments by specifying the domain, major goals, and specific 
objectives to be served^ Evaluative information, such as needs 
assessment data and demographic information, aids in making these 
types of decisions. 

Programming .decisions specify procedures, personnel, facili- 
ties, resources, and time requirements for establishing and plan- 
ning a program. These types of decisions are obviously also a 
type of planning decision. 

Implementation decisions include those that guide the imple- 
mentation or installation of a program activity. Evaluative 
information is useful in this type of decision, which can be 
thought of as primarily involving program improvement. 

Finally, recycling decisions deal with the alternatives of 
terminating, sustaining, adjusting, or drastically modifying an 
educational program. The information gained from evaluation is 
important in deciding program priorities and allocations of 
resources, since not all activities can be carried on or expanded 
indefinitely. 

Most of these decisions involve some type of change. Each 
evaluation report, whether a needs assessment report or a compre- 
hensive program report, can provide an interpretation of a prob- 
lem or a need. Sample 9 presents a pictorial view of a sequence 
of events that leads to rational decision making and program 
improvement. 



4« Phi Delta Kappa National Study Committee on Evaluation: 
Daniel L. Stufflebeam et al.. Educational Evaluation and 
Decision-Making (Itasca, IL: F. E. Peacock Publisher, Inc. , 
1971). 
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SAMPLE 9 

RATIONAL DECISION-MAKING PROCESS 







Diagnose and 
Interpret Needs 





Assess Alternatives and 
Select Best Solution 



Search for Alternative 
Solutions or Actions 




Implement Best Solution 
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The loop in sample 9 starts with the evaluative data and 
aoes next to interpretation. These two steps are delineated 
in de^lil within previous sections of this information sheet. 

third Itep ?n?olves the search for alternative solutions 
or actions thS t can correct the identified problem or improve 
?Se deficient situation. This step begins with an analysis 
of existing resources. A logical starting place is to involve 
?he expertise, experiences, and knowledge of existing staff. 
Br!ins?or^ing' sessions and open discussions can o ten uncover 
excellent ideas for the remedy of identified problems. Other 
Activities that can extend staff knowledge of other solutions 
or resources include the following: 

• Attendance at professional meetings 

• Visits to other agencies 

• Phone calls to outsiders 

Once a bank of alternatives has been acquired, the f£U|th 
step sEotld involve the assessment of alternatives and the selec- 
ItS of one solution for implementation. This step is mostly 
baled on cLmon sense; however, there are several techniques that 
cJn improve the process. These include (1) acquiring an overview 
or descrirtion o? each alternative from a --P-^---%^^^,^'t 
oniirrP (2) contacting at least one person who has had direct 
Ixperienie with the alternative, (3) observing the alternative in 
S rn%nother setting, (4) obtaining evaluative data o^ 
alternative from another source, (5) pilot testing tne ^-^^ernd 
tive Snd te) involving appropriate others in the selection of 

the best alternative. Any of these techniques will help in 
better understanding an alternative and lead to ^ ^^"^^"J^^^tly 

rational-based decision. Involving Pf'^^o"^ "J^^^^e^i^on is 

affected by or concerned with the implementation of a decision is 

always a wise management decision. 

The fifth step in sample 9 is the implementation of the best 
alternati ve in the educational program. This involves informing 
staff" delegating responsibility, and monitoring implementation. 

afterTt H^^.^^e^^ntr ^ ^^li^:^^"^^^ 
SMi--^^^ -HLrv^?!"ft ^ly^-^poss^b-e^to-ifpr pe 

ZS^'l^^^^-^^ T-ifbunt^i? 

neS prSgta; plan^, underscores the implementation P^^^^^^^' f 
sures p?ogram resilts, suggests needed changes, and is built 
into the plans for carrying out ^^ose changes. t is, in^ 
not a device for accumulating a data bank P^^/^' ^^^^i^ility as 
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Two excellent references that you may wish to review 
^ if you want more in-depth information on the report- 
Optional ^^9 of evaluation data are Morris and Fitz-Gibbon, 
Activity ■ How to Present an Evaluation Report ; and U.S. Office 
Ql: Education, Preparing Evaluation Reports; A Guid e 
— for Authors . ' " 

How to Present an Evaluation Report is an 80-page 
paperback designed "to help you convey the evaluation 
information you have collected to your various audi- 
ences as effectively and painlessly as possible." It 
is simply and clearly written, contains help in orga- 
nizing your thoughts, provides an evaluation report 
outline with explanations of each section of the out- 
line, gives practical tips about preparing different 
types of reports for different audiences, and pro- 
vides information on using graphs and tables to pre- 
sent data. 



Preparing Evaluation Reports; A Guide for Authors is 
a government-prepared 68-page paperback to help per- 
sons preparing evaluation reports for educational 
programs. Four sections provide assistance in 

(1) what to include to describe the context or back- 
ground (e.g., local demographics and school system 
characteristics) of the program being evaluated, 

(2) how to describe the program itself, (3) how to 
report evidence of changes brought about by the pro- 
gram, and (4) how to derive and report recommenda- 
tions. In addition, there is a five-page listing of 
related references, grouped by difficulty level 
(easy, harder, difficult), and covering (1) research 
methodology and experimental design, (2) sampling, 

(3) measurement: test theory, (4) measurement; test 
construction, (5) measurement: periodicals rele- 
vant to educational testing, (6) analyzing data, and 
(7) data processing. Finally, the reference includes 
a ten-page sample of a narrative evaluation report. 
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The following "Case Situation" briefly describes the 
findings from a program evaluation conducted at a 
community college. Read the situation and findings 
described and (1) develop a list of conclusions that 
help to explain the information presented, and 
(2) briefly describe the recommendations you would 
make based on your conclusions. 



CASE SITUATION 



The dean of career programs at Danville Area Community Col- 
lege has recently directed an evaluation of the computer tech- 
nology program within the business department. A student 
follow-up survey, an employer survey, a survey of current stu- 
dents, and a polling of advisory committee members were conducted 
as part of the evaluation. The following fragments of data 
represent part of the findings: 



Student Follow-up (120 were surveyed; 80 responded) 

• 80 percent of the graduates were placed 

• 90 percent of the graduates felt the program was 
relevant 

• 65 percent of graduates thought the course covering 
equipment was out of date 

Employer Survey (34 were contacted; 26 replied) 

• 100 percent of responding employers were generally 
satisfied with the training of graduates 

• 18 employers indicated, in comments, that graduates were 
not familiar with their data-entry equipment 
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SA 


A 


p 


SD 


70% 


10% 


20% 


'^% 


90% 


10% 


0% 


0% 


60% 


liD% 


30% 


0% 



Current Student Survey 

The following items related co instructional r.terials: 

14. The instructional equipment in 
this program is current, 

15. The software packages are 
adequate • 

16. The textbooks used in this 
program are current. 

Advisory Committee Poll 

Advisory committee members made the following comments: 

1. The program is well administered. 

2. The curriculum is ,urrent. 

3. Some equipment is obsolete. 
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compare your written conclusions and recommendations 
with the "Model Conclusions/Recommendations" given 
below. Your responses need not exactly duplicate the 
model; however, you should should have covered the 
same major points. 



MODEL CONCLUSIONS/RECOMMENDATIONS 



Conclusions 

1. Based upon student follow-up, employer survey, and advisory 
committee poll, the equipment used in some parts of the pro- 
gram is becoming outdated. \ small percentage of current 
students also identified this as a problem. 

2. A sizable percentage of current students feel that the text- 
books used are not current, but this is not reflected in the 
responses by those in the real world of work: employers, 
graduates, and advisory committee members. 

3 According to all four sources, the program seems, overall, 
to be reasonably effective: placement is high; current stu- 
dents, graduates, and employers are generally satisfied; and 
advisory committee members feel the program is well adminis- 
tered. 

Recommendations 

1. Make program staff aware of the overall strengths of theij^^ 
program, a faot that provides a firm base from which to ^e 
improvements . 

2 Obtain additional information concerning the adequacy of the 
textbooks. Are th-.se texts a part of the course on equip- 
ment— a course that graduates thought was out of date? Have 
employers and advisory committee members reviewed these 
texts, and if so, what were their views concerning their 
adequacy? 

3. update the curriculum for the course covering equipment, 

based on a recent analysis of the skills required on the 30b 
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4. 



Purchase more up-to-date equipment (e.g., data-entry equip- 
ment). If funding is not presently available, add activi- 
ties to the curriculum to provide students with experience 
m working with such equipment through other means (e.g., a 
workshop conducted in a local business or computer equipment 
supply company, or on-the-job training). 



^Zl t Performance; Your completed conclusions and recommenda- 
Jin^io^SU ^""^ covered the same major points as the model, if 
you missed some points or have questions about any additional 
points you made, review the material in the information sheet, 
"Using Evaluation Data," pp. 57-67, or check with yoj? resource 
person if necessary. 



Learning Experience IV 

FINAL EXPERIENCE 



While working in an actual administrative situ- 
ation, direct program evaluation.* 



Activity 



As part of your administrative duties, direct a 
program evaluation effort. This will include — 

• selecting a small program or program aspect 
that merits evaluation 

• preparing or directing the preparation of 
an evaluation plan 

• gaining approval for the plan 

• involving appropriate groups in the evalu- 
ation process 

• collecting or arranging for the collection 
of data 

• interpreting or arranging for the interpre- 
tation of the evaluation results 

• preparing or directing the preparation of 
an evaluation report ( s ) 

• using the evaluation data in decision 
making 

NOTE: As you complete each of the above activi- 
ties, document your activities (in writing, on 
tape, through a log) for assessment purposes. 



continued 



If you are not currently working in an actual administrative 
situation, this learning experience may be deferred, with the 
approval of your resource person, until you have access to an 
actual administrative situation . 
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FINAL EXPERIENCE_continued 



Your resource person may want you to submit your 
written evaluation plan for review before you 
proceed with your actual evaluation efforts. 

Due to the nature of this experience, you will 
need access to an actual administrative situation 
over an extended period of time (e.g., three to 
twelve months ) . 




Arrange to have your resource person review your 
plan, data, evaluation report, and any other docu- 
mentation you have produced. If possible, arrange 
to have your resource person observe your perfor- 
mance at a point when you are directly involved in 
the process (e.g., orienting others to their roles 
and responsibilities, directing the development of 
an evaluation plan on a small-group basis). 

Your total competency will be assessed by your 
resource person, using the "Administrator Perfor- 
mance Assessment Form," pp* 75-77. 

Based upon the criteria specified in this assess- 
ment instrument, your resource perscpn will deter- 
mine whether you are competent in directing 
program evaluation. 
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Name 
Date 



ADMINISTRATOR PERFORMANCE ASSESSMENT FORM 

Direct Program Evaluation 



Directions : Indicate the level of the administrator's accom- 
plishment by placing an X in the appropriate box under the LEVEL 
OF PERFOPMANCE heading. If, because of special circumstances, a 
performance component was not applicable, or impossible to exe- 
cute, place an X in the ^VA box. 

LEVEL OF PERFORMANCE 
^o^* 9<f^' GO^^ 



7. *-easonable component or program 
vvc . 'eler/ed for evaluat .on. 



1 9 . . . 



Thi- administrator assr-ied an 
t:?j;4.ropr iate degree o£ responsi- 
bL ity for the deve}r;pment of an 
ev. lua tion plan. ... * « < 

V'd completed ev. luaticn plan 

ncluded the foil owing appropri- 
ately stated Bi^^c complete compo- 
nents : 



□ □ n □ p □ 

r 

□ □□□on 



a. 


purpose r ati:^ "*r.t« » » - - 


□ 


□ 


□ 


□ 


P 


□ 


b. 


scope s tato.i 1^*''' t .^^.r.. .r... 


.. □ 


□ 


□ 


n 




r'-j 


c . 




.. □ 


□ 


□ 


□ 


b 


□ 


d. 


activity. , information 




□ 


□ 


□ 




□ 


e. 






□ 


□ 


□ 




□ 



names of persons responsible 
far specific task assignments. 



□ □ □ □ fa □ 
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LEVEL OF PERFORMANCE 
^o<^* 60*^ ^+*^'^* 
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4. The scope of the planned evalu- 
ation is realistic (e.g., time — 
involved, resources available) ••• • | | 

5. The evaluation plan is internally 
consistent (i.e., the activities, 
infcnv^^tion sources, and instru- 
ments selected are appropriate 

me^.ns for responding to the key 

qa-iStions) I — I 

"he administrator gained approval 

ror the plan from appropriate — 

individuals and groups | — I 

7. Appropriate data-collection tech- — 
niques were selected and used | — | 

3« The administrator assumed an 

appropriate degree of responsi- 
bility for: 

a. the collection of data d] 

b. the interpretation of data.... D 

c. the preparation of an evalu- — 
ation report(s) I — I 

9. The completed evaluation report(s) 
included the following appropri- 
ately stated and complete compo- 
nents : 

a. purpose and scope statement... 

b. description of procedures — 
followed I — I 

c. description of evaluation 

methodology used I — I 

d. evaluation findings. . • • □ 

e. conclusions and recommenda- 

tions I — 1 
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□ □ □ 



□ □ 



□ □ □ 



□ □ □ □ □ 



□ □ 



□ □ □ 



□ □ 



□ □ □ p □ 

□ □ □ o □ 



□ □ □ 



□ □ 



r 

□ □□[□□ 

□ □□[□□ 

□ □ □ !□ □ 



□ □ □ 



□ □ 



LEVEL OF PERFORMANCE 



^o<^* 9<fi' V^"^' 



10. The procedures and methodology 
used were consistent with the 
original evaluation pi an 

11. The findings, conclusions, and 
recommendations are consistent 
with the raw data collected 

12. Each evaluation report answers 
satisfactorily the key questions 
delineated in the evaluation plan. 

13. The administrator involved the 
following persons or groups 
throughout the evaluation pro- 
cess (planning, implementing, 
interpreting , reporting ) as 
appropriate : 

a . instructors/teachers 

b. advisory group(s) 

c. administrators 

d. support personnel 

e. students » < 

f . external evaluators 

g. community organizations , 

14. The administrator involved others 
and used the evaluation data in 
decision making • 



□ □□□[□□ 

i 

□ □□□[□□ 

f 
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□ 
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□ 
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□ 
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□ 


□ 


□ 


□ 




□ 


□ 


□ 


□ 


□ 


;'□ 

f 


□ 


□ 


□ 


□ 




□ 



□ □□□[□□ 



Level of Perforn;ance ; All items must receive N/A, GOOD, or 
EXCELLENT responses. If any item receives a NONE, POOR, or FAIR 
response, the administrator and resource person should meet to 
determine what additional activities the administrator needs to 
complete in order to reach competency in the weak area(s). 
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COMPETENCY-BASED VOCATIONAL EDUCATION ADMINISTRATOR 

MODULE SERIES 



Order No. Module Title 

LT 58B-1 Organize and Work with a Local Vocational Education Advisory Council 

LT 58B-2 Supervise Vocational Education Personnel 

LT 58B-3 Appraise the Personnel Development Needs of Vocational Teachers 

LT 588-4 Esu^blish a Student Placement Service and Coordinate Follow-up Studies 

LT 58B-5 Develop Local Plans for Vocational Education: Part ( 

LT 58B-6 Develop Local Plans for Vocational Education: Part II 

LT 588-7 Direct Curriculum Development 

LT 58B-8 Guide the Development and Improvement of Instruction 

LT 58B-9 Promote the Vocational Education Program 

LT 58B-10 Direct Program Evaluation 

LT 58B-11 Manage Student Recruitment and Admissions 

LT 588-12 Provide a Staff Development Program 



OTHER MODULES IN PROGRESS 

Additional modules are being developed through the Consortium for the Development of 
Professional Materials for Vocational Education. The Consortium is supported by the 
following member states: Florida, Illinois. Ohio. New York. North Carolina, and 
Pennsylvania. 



RELATED MATERIALS 

LT 58A Guide to Using Competency-Based Vocational Education Administrator 
Materials 

RD 141 The Identification and National Verification of Competencies Important to 
Secondary and Post-Secondary Administrators of Vocational Education 

RD 142 The Development of Competency-Based Instructional Materials for the 
Preparation of Local Administrators of Secondary and Post-Secondary 
Vocational Education 



For information regarding availability and prices of these materials contact— 



Program Information Office 

The National Center for Research 

in Vocational Education 
The Ohio State University 
1960 Kenny Road 
Columbus. Ohio 43210 
(614) 486-3655 
(800) 848-4815 




Th8 Ohio State University 



